2022-12-01T10:00:26.9528792Z Requested labels: linux.2xlarge 2022-12-01T10:00:26.9528867Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/89997/merge 2022-12-01T10:00:26.9528889Z Waiting for a runner to pick up this job... 2022-12-01T10:00:27.3405905Z Job is about to start running on the runner: i-092913c5169eb9727 (organization) 2022-12-01T10:00:31.7584797Z Current runner version: '2.299.1' 2022-12-01T10:00:31.7590060Z Runner name: 'i-092913c5169eb9727' 2022-12-01T10:00:31.7590601Z Runner group name: 'Default' 2022-12-01T10:00:31.7591178Z Machine name: 'ip-10-0-1-92' 2022-12-01T10:00:31.7593139Z ##[group]GITHUB_TOKEN Permissions 2022-12-01T10:00:31.7593815Z Actions: read 2022-12-01T10:00:31.7594130Z Checks: read 2022-12-01T10:00:31.7594399Z Contents: read 2022-12-01T10:00:31.7594725Z Deployments: read 2022-12-01T10:00:31.7595076Z Discussions: read 2022-12-01T10:00:31.7595346Z Issues: read 2022-12-01T10:00:31.7595679Z Metadata: read 2022-12-01T10:00:31.7596001Z Packages: read 2022-12-01T10:00:31.7596270Z Pages: read 2022-12-01T10:00:31.7596608Z PullRequests: read 2022-12-01T10:00:31.7596970Z RepositoryProjects: read 2022-12-01T10:00:31.7597281Z SecurityEvents: read 2022-12-01T10:00:31.7597611Z Statuses: read 2022-12-01T10:00:31.7597919Z ##[endgroup] 2022-12-01T10:00:31.7601028Z Secret source: None 2022-12-01T10:00:31.7601794Z Prepare workflow directory 2022-12-01T10:00:32.4512788Z Prepare all required actions 2022-12-01T10:00:32.4681465Z Getting action download info 2022-12-01T10:00:32.6701536Z Download action repository 'pytorch/pytorch@master' (SHA:850b53bbee82fb194af85b566aedee94b96def32) 2022-12-01T10:00:35.7709057Z Download action repository 'pytorch/test-infra@main' (SHA:1f415583bdcd967e33ea8fd05be71ed0bdf19880) 2022-12-01T10:00:36.0079990Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767) 2022-12-01T10:00:36.1221135Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2022-12-01T10:00:36.4002532Z Getting action download info 2022-12-01T10:00:36.5711833Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076) 2022-12-01T10:00:36.7675098Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml 2022-12-01T10:00:36.7676716Z ##[group] Inputs 2022-12-01T10:00:36.7677034Z build-environment: linux-focal-py3.7-gcc7 2022-12-01T10:00:36.7677834Z test-matrix: { include: [ { config: "default", shard: 1, num_shards: 2, runner: "linux.2xlarge" }, { config: "default", shard: 2, num_shards: 2, runner: "linux.2xlarge" }, { config: "distributed", shard: 1, num_shards: 2, runner: "linux.2xlarge" }, { config: "distributed", shard: 2, num_shards: 2, runner: "linux.2xlarge" }, { config: "functorch", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "docs_test", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "backwards_compat", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, ]} 2022-12-01T10:00:36.7678851Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b 2022-12-01T10:00:36.7679214Z sync-tag: 2022-12-01T10:00:36.7679423Z ##[endgroup] 2022-12-01T10:00:36.7679934Z Complete job name: linux-focal-py3.7-gcc7 / test (distributed, 1, 2, linux.2xlarge) 2022-12-01T10:00:36.8373498Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2022-12-01T10:00:36.8373812Z with: 2022-12-01T10:00:36.8374011Z submodules: recursive 2022-12-01T10:00:36.8374222Z fetch-depth: 0 2022-12-01T10:00:36.8374419Z env: 2022-12-01T10:00:36.8374610Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:00:36.8374831Z ##[endgroup] 2022-12-01T10:00:36.8587479Z ##[group]Run retry () { 2022-12-01T10:00:36.8587739Z retry () { 2022-12-01T10:00:36.8587996Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2022-12-01T10:00:36.8588236Z } 2022-12-01T10:00:36.8588445Z echo "${GITHUB_WORKSPACE}" 2022-12-01T10:00:36.8588677Z if [ -z "${NO_SUDO}" ]; then 2022-12-01T10:00:36.8588945Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2022-12-01T10:00:36.8589207Z else 2022-12-01T10:00:36.8589421Z  retry rm -rf "${GITHUB_WORKSPACE}" 2022-12-01T10:00:36.8589778Z fi 2022-12-01T10:00:36.8590022Z mkdir "${GITHUB_WORKSPACE}" 2022-12-01T10:00:36.8605239Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:00:36.8605510Z env: 2022-12-01T10:00:36.8605718Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:00:36.8605919Z NO_SUDO: 2022-12-01T10:00:36.8606118Z ##[endgroup] 2022-12-01T10:00:36.8771437Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-12-01T10:00:39.5774128Z ##[group]Run malfet/checkout@silent-checkout 2022-12-01T10:00:39.5774363Z with: 2022-12-01T10:00:39.5774565Z ref: c13d400bffe90e16b96520bbc8a41a6f0c9cd584 2022-12-01T10:00:39.5774775Z fetch-depth: 0 2022-12-01T10:00:39.5774946Z submodules: recursive 2022-12-01T10:00:39.5775131Z quiet-checkout: true 2022-12-01T10:00:39.5775332Z repository: pytorch/pytorch 2022-12-01T10:00:39.5775692Z token: *** 2022-12-01T10:00:39.5775862Z ssh-strict: true 2022-12-01T10:00:39.5776055Z persist-credentials: true 2022-12-01T10:00:39.5776253Z clean: true 2022-12-01T10:00:39.5776413Z lfs: false 2022-12-01T10:00:39.5776593Z set-safe-directory: true 2022-12-01T10:00:39.5776771Z env: 2022-12-01T10:00:39.5776927Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:00:39.5777109Z ##[endgroup] 2022-12-01T10:00:39.6915031Z Syncing repository: pytorch/pytorch 2022-12-01T10:00:39.6916789Z ##[group]Getting Git version info 2022-12-01T10:00:39.6917493Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-12-01T10:00:39.6918035Z [command]/usr/bin/git version 2022-12-01T10:00:39.6918328Z git version 2.37.1 2022-12-01T10:00:39.6918896Z ##[endgroup] 2022-12-01T10:00:39.6930214Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/5d138f43-ce94-4be9-be37-44811d3acd12' before making global git config changes 2022-12-01T10:00:39.6930930Z Adding repository directory to the temporary git global config as a safe directory 2022-12-01T10:00:39.6934885Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-12-01T10:00:39.6971244Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-12-01T10:00:39.6975338Z ##[group]Initializing the repository 2022-12-01T10:00:39.6977630Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-12-01T10:00:39.7127797Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-12-01T10:00:39.7128350Z hint: is subject to change. To configure the initial branch name to use in all 2022-12-01T10:00:39.7128949Z hint: of your new repositories, which will suppress this warning, call: 2022-12-01T10:00:39.7129378Z hint: 2022-12-01T10:00:39.7129871Z hint: git config --global init.defaultBranch 2022-12-01T10:00:39.7130395Z hint: 2022-12-01T10:00:39.7130933Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-12-01T10:00:39.7131630Z hint: 'development'. The just-created branch can be renamed via this command: 2022-12-01T10:00:39.7132051Z hint: 2022-12-01T10:00:39.7132428Z hint: git branch -m 2022-12-01T10:00:39.7132804Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2022-12-01T10:00:39.7137570Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-12-01T10:00:39.7164093Z ##[endgroup] 2022-12-01T10:00:39.7164667Z ##[group]Disabling automatic garbage collection 2022-12-01T10:00:39.7167222Z [command]/usr/bin/git config --local gc.auto 0 2022-12-01T10:00:39.7191119Z ##[endgroup] 2022-12-01T10:00:39.7192019Z ##[group]Setting up auth 2022-12-01T10:00:39.7197476Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-12-01T10:00:39.7224053Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-12-01T10:00:39.7459109Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-12-01T10:00:39.7486705Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-12-01T10:00:39.7721307Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-12-01T10:00:39.7757389Z ##[endgroup] 2022-12-01T10:00:39.7757961Z ##[group]Fetching the repository 2022-12-01T10:00:39.7763909Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-12-01T10:01:31.6919451Z [command]/usr/bin/git rev-parse --verify --quiet c13d400bffe90e16b96520bbc8a41a6f0c9cd584^{object} 2022-12-01T10:01:31.6955321Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin c13d400bffe90e16b96520bbc8a41a6f0c9cd584 2022-12-01T10:01:32.9539353Z ##[endgroup] 2022-12-01T10:01:32.9539908Z ##[group]Determining the checkout info 2022-12-01T10:01:32.9540738Z ##[endgroup] 2022-12-01T10:01:32.9541057Z ##[group]Checking out the ref 2022-12-01T10:01:32.9544716Z [command]/usr/bin/git checkout --quiet --force c13d400bffe90e16b96520bbc8a41a6f0c9cd584 2022-12-01T10:01:34.1722906Z ##[endgroup] 2022-12-01T10:01:34.1723459Z ##[group]Setting up auth for fetching submodules 2022-12-01T10:01:34.1728544Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-12-01T10:01:34.1776514Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-12-01T10:01:34.1805267Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2022-12-01T10:01:34.1832398Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2022-12-01T10:01:34.1855792Z ##[endgroup] 2022-12-01T10:01:34.1856138Z ##[group]Fetching submodules 2022-12-01T10:01:34.1859277Z [command]/usr/bin/git submodule sync --recursive 2022-12-01T10:01:34.2114103Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-12-01T10:01:34.2361861Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-12-01T10:01:34.2363023Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-12-01T10:01:34.2365290Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-12-01T10:01:34.2367647Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-12-01T10:01:34.2370153Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-12-01T10:01:34.2373061Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2022-12-01T10:01:34.2375530Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-12-01T10:01:34.2378377Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-12-01T10:01:34.2381461Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-12-01T10:01:34.2384480Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-12-01T10:01:34.2387754Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-12-01T10:01:34.2390870Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2022-12-01T10:01:34.2394188Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2022-12-01T10:01:34.2397654Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-12-01T10:01:34.2401544Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-12-01T10:01:34.2405117Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-12-01T10:01:34.2408873Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-12-01T10:01:34.2413012Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-12-01T10:01:34.2416827Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-12-01T10:01:34.2421017Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-12-01T10:01:34.2425156Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-12-01T10:01:34.2429634Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-12-01T10:01:34.2433959Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2022-12-01T10:01:34.2438593Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-12-01T10:01:34.2443203Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-12-01T10:01:34.2447887Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-12-01T10:01:34.2452612Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2022-12-01T10:01:34.2457491Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-12-01T10:01:34.2462625Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-12-01T10:01:34.2467679Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-12-01T10:01:34.2472871Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-12-01T10:01:34.2478131Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-12-01T10:01:34.2486485Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-12-01T10:01:34.2492321Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-12-01T10:01:34.2498438Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-12-01T10:01:34.2504840Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-12-01T10:01:34.2511230Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-12-01T10:01:34.2517658Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-12-01T10:01:34.2524478Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-12-01T10:01:34.2531198Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-12-01T10:01:34.2538084Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-12-01T10:01:34.2561618Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-12-01T10:01:34.5426588Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-12-01T10:01:34.7659243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-12-01T10:01:34.9750766Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-12-01T10:01:35.2828365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-12-01T10:01:35.5793920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2022-12-01T10:01:37.6153581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-12-01T10:01:43.4642305Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-12-01T10:01:43.8406180Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-12-01T10:01:44.4062562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-12-01T10:01:45.8619051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-12-01T10:01:46.9769298Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2022-12-01T10:01:48.3668899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-12-01T10:01:53.5835548Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-12-01T10:01:54.4218030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-12-01T10:01:55.6272061Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-12-01T10:01:56.8299608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-12-01T10:01:57.0430740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-12-01T10:01:57.5182742Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-12-01T10:01:57.8302005Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-12-01T10:01:58.8897077Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-12-01T10:01:59.3385540Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-12-01T10:01:59.5355959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2022-12-01T10:01:59.8313785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-12-01T10:02:01.3946623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-12-01T10:02:02.0050299Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-12-01T10:02:02.8529266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2022-12-01T10:02:08.8899590Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-12-01T10:02:10.5956606Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-12-01T10:02:11.0954352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-12-01T10:02:11.3599371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-12-01T10:02:16.8140226Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-12-01T10:02:17.1803238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-12-01T10:02:17.4412411Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-12-01T10:02:18.2139236Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-12-01T10:02:18.4429690Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-12-01T10:02:18.7741289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-12-01T10:02:19.0599344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-12-01T10:02:19.6255837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-12-01T10:02:21.7718636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-12-01T10:02:22.2657365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-12-01T10:02:24.4360159Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-12-01T10:02:24.4456078Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-12-01T10:02:24.4530672Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-12-01T10:02:24.4735192Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-12-01T10:02:24.4933726Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-12-01T10:02:24.5252483Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2022-12-01T10:02:25.0720955Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-12-01T10:02:25.0912135Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-12-01T10:02:25.1812792Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2022-12-01T10:02:25.2116206Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-12-01T10:02:25.4733697Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3' 2022-12-01T10:02:25.8435307Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2022-12-01T10:02:26.0642837Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2022-12-01T10:02:26.1059391Z Submodule path 'third_party/fbgemm': checked out '0d98c261561524cce92e37fe307ea6596664309a' 2022-12-01T10:02:26.1072112Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-12-01T10:02:26.1073989Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-12-01T10:02:26.1076050Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-12-01T10:02:26.1078494Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2022-12-01T10:02:26.1099949Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-12-01T10:02:27.0645069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-12-01T10:02:27.7795114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-12-01T10:02:28.8236796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2022-12-01T10:02:29.1478425Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2022-12-01T10:02:29.2384252Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-12-01T10:02:29.2915551Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-12-01T10:02:29.3006447Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2022-12-01T10:02:29.3758911Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-12-01T10:02:29.4074287Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2022-12-01T10:02:29.4154185Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-12-01T10:02:29.4509472Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-12-01T10:02:29.4726521Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72' 2022-12-01T10:02:29.5144894Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-12-01T10:02:29.5247989Z Submodule path 'third_party/ideep': checked out '77d662b313a762e82b389d3fd965e0098f12cd99' 2022-12-01T10:02:29.5261172Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-12-01T10:02:29.5281132Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-12-01T10:02:37.0400238Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46' 2022-12-01T10:02:37.0415431Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-12-01T10:02:37.0437863Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-12-01T10:02:44.6810874Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815' 2022-12-01T10:02:44.6905624Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-12-01T10:02:44.7032953Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2022-12-01T10:02:44.7896170Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573' 2022-12-01T10:02:44.7911660Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-12-01T10:02:44.7913557Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-12-01T10:02:44.7935386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-12-01T10:02:45.9731805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-12-01T10:02:47.0570497Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-12-01T10:02:47.1064859Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-12-01T10:02:47.1246518Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2022-12-01T10:02:47.1370738Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-12-01T10:02:47.2342388Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2022-12-01T10:02:47.4571199Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2022-12-01T10:02:47.4597329Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-12-01T10:02:47.4598762Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-12-01T10:02:47.4621631Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-12-01T10:02:47.8574752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-12-01T10:02:48.6537323Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-12-01T10:02:48.6816878Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2022-12-01T10:02:48.6954271Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-12-01T10:02:48.6966772Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-12-01T10:02:48.6987453Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-12-01T10:02:50.4307985Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-12-01T10:02:50.4327181Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-12-01T10:02:50.4328740Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-12-01T10:02:50.4351021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-12-01T10:02:50.8227280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-12-01T10:02:51.6024452Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-12-01T10:02:51.6615472Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-12-01T10:02:51.6628248Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-12-01T10:02:51.6649199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-12-01T10:02:51.8884610Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-12-01T10:02:51.8966570Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-12-01T10:02:52.1293553Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-12-01T10:02:52.1311679Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-12-01T10:02:52.1313532Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-12-01T10:02:52.1338084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-12-01T10:02:52.5334477Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-12-01T10:02:53.6342960Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-12-01T10:02:53.6961111Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-12-01T10:02:53.7037869Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-12-01T10:02:53.7133088Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-12-01T10:02:53.7432228Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020' 2022-12-01T10:02:53.7510683Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-12-01T10:02:53.7759953Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2022-12-01T10:02:53.7849529Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-12-01T10:02:53.8284268Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-12-01T10:02:53.9260876Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-12-01T10:02:53.9494266Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-12-01T10:02:53.9507255Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-12-01T10:02:53.9509475Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-12-01T10:02:53.9511927Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-12-01T10:02:53.9514434Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-12-01T10:02:53.9536050Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-12-01T10:02:54.9917319Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-12-01T10:02:55.2821764Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-12-01T10:02:56.5768982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-12-01T10:02:57.3998626Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-12-01T10:02:57.4127646Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-12-01T10:02:57.4713015Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-12-01T10:02:57.4957721Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-12-01T10:02:57.4969716Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-12-01T10:02:57.4990657Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-12-01T10:02:57.7727642Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-12-01T10:02:57.8948282Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-12-01T10:02:57.8974753Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-12-01T10:02:57.9238444Z Entering 'android/libs/fbjni' 2022-12-01T10:02:57.9272753Z Entering 'third_party/FP16' 2022-12-01T10:02:57.9307331Z Entering 'third_party/FXdiv' 2022-12-01T10:02:57.9343381Z Entering 'third_party/NNPACK' 2022-12-01T10:02:57.9378979Z Entering 'third_party/QNNPACK' 2022-12-01T10:02:57.9412967Z Entering 'third_party/VulkanMemoryAllocator' 2022-12-01T10:02:57.9449664Z Entering 'third_party/XNNPACK' 2022-12-01T10:02:57.9495865Z Entering 'third_party/benchmark' 2022-12-01T10:02:57.9532506Z Entering 'third_party/cpuinfo' 2022-12-01T10:02:57.9567405Z Entering 'third_party/cub' 2022-12-01T10:02:57.9600519Z Entering 'third_party/cudnn_frontend' 2022-12-01T10:02:57.9638899Z Entering 'third_party/cutlass' 2022-12-01T10:02:57.9678679Z Entering 'third_party/eigen' 2022-12-01T10:02:57.9715276Z Entering 'third_party/fbgemm' 2022-12-01T10:02:57.9752600Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-12-01T10:02:57.9788361Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-12-01T10:02:57.9824548Z Entering 'third_party/fbgemm/third_party/googletest' 2022-12-01T10:02:57.9862608Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-12-01T10:02:57.9900753Z Entering 'third_party/flatbuffers' 2022-12-01T10:02:57.9937805Z Entering 'third_party/fmt' 2022-12-01T10:02:57.9970990Z Entering 'third_party/foxi' 2022-12-01T10:02:58.0006358Z Entering 'third_party/gemmlowp/gemmlowp' 2022-12-01T10:02:58.0039918Z Entering 'third_party/gloo' 2022-12-01T10:02:58.0073176Z Entering 'third_party/googletest' 2022-12-01T10:02:58.0106630Z Entering 'third_party/ideep' 2022-12-01T10:02:58.0139707Z Entering 'third_party/ideep/mkl-dnn' 2022-12-01T10:02:58.0173394Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-12-01T10:02:58.0216171Z Entering 'third_party/ios-cmake' 2022-12-01T10:02:58.0250173Z Entering 'third_party/ittapi' 2022-12-01T10:02:58.0283858Z Entering 'third_party/kineto' 2022-12-01T10:02:58.0316669Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-12-01T10:02:58.0351788Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-12-01T10:02:58.0387758Z Entering 'third_party/nccl/nccl' 2022-12-01T10:02:58.0422418Z Entering 'third_party/neon2sse' 2022-12-01T10:02:58.0457392Z Entering 'third_party/nlohmann' 2022-12-01T10:02:58.0492338Z Entering 'third_party/onnx' 2022-12-01T10:02:58.0538346Z Entering 'third_party/onnx/third_party/benchmark' 2022-12-01T10:02:58.0572630Z Entering 'third_party/onnx/third_party/pybind11' 2022-12-01T10:02:58.0609656Z Entering 'third_party/onnx-tensorrt' 2022-12-01T10:02:58.0642917Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-12-01T10:02:58.0679463Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-12-01T10:02:58.0712644Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-12-01T10:02:58.0745798Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-12-01T10:02:58.0784035Z Entering 'third_party/pocketfft' 2022-12-01T10:02:58.0819472Z Entering 'third_party/protobuf' 2022-12-01T10:02:58.0858571Z Entering 'third_party/protobuf/third_party/benchmark' 2022-12-01T10:02:58.0893256Z Entering 'third_party/protobuf/third_party/googletest' 2022-12-01T10:02:58.0932689Z Entering 'third_party/psimd' 2022-12-01T10:02:58.0970431Z Entering 'third_party/pthreadpool' 2022-12-01T10:02:58.1005715Z Entering 'third_party/pybind11' 2022-12-01T10:02:58.1041555Z Entering 'third_party/python-enum' 2022-12-01T10:02:58.1076266Z Entering 'third_party/python-peachpy' 2022-12-01T10:02:58.1111340Z Entering 'third_party/python-six' 2022-12-01T10:02:58.1146782Z Entering 'third_party/sleef' 2022-12-01T10:02:58.1181372Z Entering 'third_party/tbb' 2022-12-01T10:02:58.1217711Z Entering 'third_party/tensorpipe' 2022-12-01T10:02:58.1252708Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-12-01T10:02:58.1287511Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-12-01T10:02:58.1321935Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-12-01T10:02:58.1356640Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-12-01T10:02:58.1390279Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-12-01T10:02:58.1426962Z Entering 'third_party/zstd' 2022-12-01T10:02:58.1468930Z ##[endgroup] 2022-12-01T10:02:58.1471404Z ##[group]Persisting credentials for submodules 2022-12-01T10:02:58.1485335Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-12-01T10:02:58.1737592Z Entering 'android/libs/fbjni' 2022-12-01T10:02:58.1771178Z Entering 'third_party/FP16' 2022-12-01T10:02:58.1804910Z Entering 'third_party/FXdiv' 2022-12-01T10:02:58.1838073Z Entering 'third_party/NNPACK' 2022-12-01T10:02:58.1871876Z Entering 'third_party/QNNPACK' 2022-12-01T10:02:58.1908281Z Entering 'third_party/VulkanMemoryAllocator' 2022-12-01T10:02:58.1940645Z Entering 'third_party/XNNPACK' 2022-12-01T10:02:58.1982916Z Entering 'third_party/benchmark' 2022-12-01T10:02:58.2017577Z Entering 'third_party/cpuinfo' 2022-12-01T10:02:58.2052190Z Entering 'third_party/cub' 2022-12-01T10:02:58.2085461Z Entering 'third_party/cudnn_frontend' 2022-12-01T10:02:58.2124763Z Entering 'third_party/cutlass' 2022-12-01T10:02:58.2163937Z Entering 'third_party/eigen' 2022-12-01T10:02:58.2198632Z Entering 'third_party/fbgemm' 2022-12-01T10:02:58.2231244Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-12-01T10:02:58.2264625Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-12-01T10:02:58.2296625Z Entering 'third_party/fbgemm/third_party/googletest' 2022-12-01T10:02:58.2328262Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-12-01T10:02:58.2361397Z Entering 'third_party/flatbuffers' 2022-12-01T10:02:58.2395595Z Entering 'third_party/fmt' 2022-12-01T10:02:58.2429513Z Entering 'third_party/foxi' 2022-12-01T10:02:58.2462559Z Entering 'third_party/gemmlowp/gemmlowp' 2022-12-01T10:02:58.2496417Z Entering 'third_party/gloo' 2022-12-01T10:02:58.2530112Z Entering 'third_party/googletest' 2022-12-01T10:02:58.2563310Z Entering 'third_party/ideep' 2022-12-01T10:02:58.2595099Z Entering 'third_party/ideep/mkl-dnn' 2022-12-01T10:02:58.2629847Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-12-01T10:02:58.2669032Z Entering 'third_party/ios-cmake' 2022-12-01T10:02:58.2702275Z Entering 'third_party/ittapi' 2022-12-01T10:02:58.2735721Z Entering 'third_party/kineto' 2022-12-01T10:02:58.2771959Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-12-01T10:02:58.2805541Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-12-01T10:02:58.2839402Z Entering 'third_party/nccl/nccl' 2022-12-01T10:02:58.2874978Z Entering 'third_party/neon2sse' 2022-12-01T10:02:58.2909090Z Entering 'third_party/nlohmann' 2022-12-01T10:02:58.2945376Z Entering 'third_party/onnx' 2022-12-01T10:02:58.2991676Z Entering 'third_party/onnx/third_party/benchmark' 2022-12-01T10:02:58.3025618Z Entering 'third_party/onnx/third_party/pybind11' 2022-12-01T10:02:58.3061103Z Entering 'third_party/onnx-tensorrt' 2022-12-01T10:02:58.3094399Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-12-01T10:02:58.3130841Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-12-01T10:02:58.3164006Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-12-01T10:02:58.3196078Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-12-01T10:02:58.3233168Z Entering 'third_party/pocketfft' 2022-12-01T10:02:58.3265453Z Entering 'third_party/protobuf' 2022-12-01T10:02:58.3300881Z Entering 'third_party/protobuf/third_party/benchmark' 2022-12-01T10:02:58.3334980Z Entering 'third_party/protobuf/third_party/googletest' 2022-12-01T10:02:58.3370144Z Entering 'third_party/psimd' 2022-12-01T10:02:58.3403436Z Entering 'third_party/pthreadpool' 2022-12-01T10:02:58.3436635Z Entering 'third_party/pybind11' 2022-12-01T10:02:58.3470786Z Entering 'third_party/python-enum' 2022-12-01T10:02:58.3504759Z Entering 'third_party/python-peachpy' 2022-12-01T10:02:58.3540375Z Entering 'third_party/python-six' 2022-12-01T10:02:58.3575594Z Entering 'third_party/sleef' 2022-12-01T10:02:58.3610558Z Entering 'third_party/tbb' 2022-12-01T10:02:58.3647694Z Entering 'third_party/tensorpipe' 2022-12-01T10:02:58.3681931Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-12-01T10:02:58.3714529Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-12-01T10:02:58.3746589Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-12-01T10:02:58.3780464Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-12-01T10:02:58.3812954Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-12-01T10:02:58.3848606Z Entering 'third_party/zstd' 2022-12-01T10:02:58.3894127Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-12-01T10:02:58.4148700Z Entering 'android/libs/fbjni' 2022-12-01T10:02:58.4179850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-12-01T10:02:58.4194369Z Entering 'third_party/FP16' 2022-12-01T10:02:58.4226080Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-12-01T10:02:58.4241536Z Entering 'third_party/FXdiv' 2022-12-01T10:02:58.4273119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-12-01T10:02:58.4287167Z Entering 'third_party/NNPACK' 2022-12-01T10:02:58.4318493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-12-01T10:02:58.4332418Z Entering 'third_party/QNNPACK' 2022-12-01T10:02:58.4364571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-12-01T10:02:58.4378559Z Entering 'third_party/VulkanMemoryAllocator' 2022-12-01T10:02:58.4409831Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2022-12-01T10:02:58.4424089Z Entering 'third_party/XNNPACK' 2022-12-01T10:02:58.4455699Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-12-01T10:02:58.4479838Z Entering 'third_party/benchmark' 2022-12-01T10:02:58.4511291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-12-01T10:02:58.4525936Z Entering 'third_party/cpuinfo' 2022-12-01T10:02:58.4557612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-12-01T10:02:58.4571855Z Entering 'third_party/cub' 2022-12-01T10:02:58.4603451Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-12-01T10:02:58.4617715Z Entering 'third_party/cudnn_frontend' 2022-12-01T10:02:58.4649168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-12-01T10:02:58.4668111Z Entering 'third_party/cutlass' 2022-12-01T10:02:58.4701022Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2022-12-01T10:02:58.4722079Z Entering 'third_party/eigen' 2022-12-01T10:02:58.4752855Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-12-01T10:02:58.4768577Z Entering 'third_party/fbgemm' 2022-12-01T10:02:58.4801730Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-12-01T10:02:58.4816313Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-12-01T10:02:58.4848151Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-12-01T10:02:58.4863523Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-12-01T10:02:58.4895449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-12-01T10:02:58.4909543Z Entering 'third_party/fbgemm/third_party/googletest' 2022-12-01T10:02:58.4941917Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-12-01T10:02:58.4955928Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-12-01T10:02:58.4986481Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2022-12-01T10:02:58.5002271Z Entering 'third_party/flatbuffers' 2022-12-01T10:02:58.5033376Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-12-01T10:02:58.5050319Z Entering 'third_party/fmt' 2022-12-01T10:02:58.5082700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-12-01T10:02:58.5097338Z Entering 'third_party/foxi' 2022-12-01T10:02:58.5129403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-12-01T10:02:58.5143868Z Entering 'third_party/gemmlowp/gemmlowp' 2022-12-01T10:02:58.5175075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-12-01T10:02:58.5189451Z Entering 'third_party/gloo' 2022-12-01T10:02:58.5220642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-12-01T10:02:58.5234502Z Entering 'third_party/googletest' 2022-12-01T10:02:58.5265647Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-12-01T10:02:58.5280936Z Entering 'third_party/ideep' 2022-12-01T10:02:58.5312985Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-12-01T10:02:58.5326733Z Entering 'third_party/ideep/mkl-dnn' 2022-12-01T10:02:58.5356916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-12-01T10:02:58.5372031Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-12-01T10:02:58.5403166Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-12-01T10:02:58.5422694Z Entering 'third_party/ios-cmake' 2022-12-01T10:02:58.5454383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-12-01T10:02:58.5469328Z Entering 'third_party/ittapi' 2022-12-01T10:02:58.5501006Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2022-12-01T10:02:58.5515492Z Entering 'third_party/kineto' 2022-12-01T10:02:58.5548843Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-12-01T10:02:58.5563285Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-12-01T10:02:58.5596923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-12-01T10:02:58.5611235Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-12-01T10:02:58.5642398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-12-01T10:02:58.5658187Z Entering 'third_party/nccl/nccl' 2022-12-01T10:02:58.5690727Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-12-01T10:02:58.5705051Z Entering 'third_party/neon2sse' 2022-12-01T10:02:58.5736839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-12-01T10:02:58.5750772Z Entering 'third_party/nlohmann' 2022-12-01T10:02:58.5782222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2022-12-01T10:02:58.5797965Z Entering 'third_party/onnx' 2022-12-01T10:02:58.5828936Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-12-01T10:02:58.5857251Z Entering 'third_party/onnx/third_party/benchmark' 2022-12-01T10:02:58.5887889Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-12-01T10:02:58.5902752Z Entering 'third_party/onnx/third_party/pybind11' 2022-12-01T10:02:58.5934557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-12-01T10:02:58.5951438Z Entering 'third_party/onnx-tensorrt' 2022-12-01T10:02:58.5983471Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-12-01T10:02:58.5996302Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-12-01T10:02:58.6027905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-12-01T10:02:58.6046071Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-12-01T10:02:58.6078008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-12-01T10:02:58.6092192Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-12-01T10:02:58.6123482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-12-01T10:02:58.6136407Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-12-01T10:02:58.6170307Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-12-01T10:02:58.6187914Z Entering 'third_party/pocketfft' 2022-12-01T10:02:58.6220334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-12-01T10:02:58.6234000Z Entering 'third_party/protobuf' 2022-12-01T10:02:58.6267901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-12-01T10:02:58.6285254Z Entering 'third_party/protobuf/third_party/benchmark' 2022-12-01T10:02:58.6316837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-12-01T10:02:58.6332257Z Entering 'third_party/protobuf/third_party/googletest' 2022-12-01T10:02:58.6361917Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-12-01T10:02:58.6377758Z Entering 'third_party/psimd' 2022-12-01T10:02:58.6408651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-12-01T10:02:58.6424107Z Entering 'third_party/pthreadpool' 2022-12-01T10:02:58.6456197Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-12-01T10:02:58.6472217Z Entering 'third_party/pybind11' 2022-12-01T10:02:58.6503490Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-12-01T10:02:58.6517963Z Entering 'third_party/python-enum' 2022-12-01T10:02:58.6548898Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-12-01T10:02:58.6563403Z Entering 'third_party/python-peachpy' 2022-12-01T10:02:58.6594832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-12-01T10:02:58.6609810Z Entering 'third_party/python-six' 2022-12-01T10:02:58.6642224Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-12-01T10:02:58.6656272Z Entering 'third_party/sleef' 2022-12-01T10:02:58.6687028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-12-01T10:02:58.6701061Z Entering 'third_party/tbb' 2022-12-01T10:02:58.6732168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-12-01T10:02:58.6748222Z Entering 'third_party/tensorpipe' 2022-12-01T10:02:58.6780839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-12-01T10:02:58.6794586Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-12-01T10:02:58.6825184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-12-01T10:02:58.6839098Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-12-01T10:02:58.6869707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-12-01T10:02:58.6884226Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-12-01T10:02:58.6914707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-12-01T10:02:58.6928394Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-12-01T10:02:58.6959066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-12-01T10:02:58.6972257Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-12-01T10:02:58.7002709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-12-01T10:02:58.7019524Z Entering 'third_party/zstd' 2022-12-01T10:02:58.7055642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-12-01T10:02:59.0076818Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-12-01T10:02:59.0340159Z Entering 'android/libs/fbjni' 2022-12-01T10:02:59.0375279Z Entering 'third_party/FP16' 2022-12-01T10:02:59.0411694Z Entering 'third_party/FXdiv' 2022-12-01T10:02:59.0447125Z Entering 'third_party/NNPACK' 2022-12-01T10:02:59.0483624Z Entering 'third_party/QNNPACK' 2022-12-01T10:02:59.0518627Z Entering 'third_party/VulkanMemoryAllocator' 2022-12-01T10:02:59.0552765Z Entering 'third_party/XNNPACK' 2022-12-01T10:02:59.0596962Z Entering 'third_party/benchmark' 2022-12-01T10:02:59.0631003Z Entering 'third_party/cpuinfo' 2022-12-01T10:02:59.0666563Z Entering 'third_party/cub' 2022-12-01T10:02:59.0700386Z Entering 'third_party/cudnn_frontend' 2022-12-01T10:02:59.0739786Z Entering 'third_party/cutlass' 2022-12-01T10:02:59.0780074Z Entering 'third_party/eigen' 2022-12-01T10:02:59.0816270Z Entering 'third_party/fbgemm' 2022-12-01T10:02:59.0851540Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-12-01T10:02:59.0884999Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-12-01T10:02:59.0920339Z Entering 'third_party/fbgemm/third_party/googletest' 2022-12-01T10:02:59.0954322Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-12-01T10:02:59.0989360Z Entering 'third_party/flatbuffers' 2022-12-01T10:02:59.1024870Z Entering 'third_party/fmt' 2022-12-01T10:02:59.1059130Z Entering 'third_party/foxi' 2022-12-01T10:02:59.1094403Z Entering 'third_party/gemmlowp/gemmlowp' 2022-12-01T10:02:59.1129264Z Entering 'third_party/gloo' 2022-12-01T10:02:59.1163672Z Entering 'third_party/googletest' 2022-12-01T10:02:59.1197408Z Entering 'third_party/ideep' 2022-12-01T10:02:59.1230572Z Entering 'third_party/ideep/mkl-dnn' 2022-12-01T10:02:59.1265931Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-12-01T10:02:59.1307214Z Entering 'third_party/ios-cmake' 2022-12-01T10:02:59.1340859Z Entering 'third_party/ittapi' 2022-12-01T10:02:59.1375996Z Entering 'third_party/kineto' 2022-12-01T10:02:59.1410543Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-12-01T10:02:59.1444015Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-12-01T10:02:59.1478414Z Entering 'third_party/nccl/nccl' 2022-12-01T10:02:59.1512961Z Entering 'third_party/neon2sse' 2022-12-01T10:02:59.1548430Z Entering 'third_party/nlohmann' 2022-12-01T10:02:59.1585362Z Entering 'third_party/onnx' 2022-12-01T10:02:59.1629559Z Entering 'third_party/onnx/third_party/benchmark' 2022-12-01T10:02:59.1663482Z Entering 'third_party/onnx/third_party/pybind11' 2022-12-01T10:02:59.1698395Z Entering 'third_party/onnx-tensorrt' 2022-12-01T10:02:59.1732182Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-12-01T10:02:59.1768917Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-12-01T10:02:59.1803014Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-12-01T10:02:59.1836962Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-12-01T10:02:59.1876369Z Entering 'third_party/pocketfft' 2022-12-01T10:02:59.1910349Z Entering 'third_party/protobuf' 2022-12-01T10:02:59.1950118Z Entering 'third_party/protobuf/third_party/benchmark' 2022-12-01T10:02:59.1983416Z Entering 'third_party/protobuf/third_party/googletest' 2022-12-01T10:02:59.2017913Z Entering 'third_party/psimd' 2022-12-01T10:02:59.2054058Z Entering 'third_party/pthreadpool' 2022-12-01T10:02:59.2088485Z Entering 'third_party/pybind11' 2022-12-01T10:02:59.2122583Z Entering 'third_party/python-enum' 2022-12-01T10:02:59.2157714Z Entering 'third_party/python-peachpy' 2022-12-01T10:02:59.2191839Z Entering 'third_party/python-six' 2022-12-01T10:02:59.2226447Z Entering 'third_party/sleef' 2022-12-01T10:02:59.2260576Z Entering 'third_party/tbb' 2022-12-01T10:02:59.2296495Z Entering 'third_party/tensorpipe' 2022-12-01T10:02:59.2331504Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-12-01T10:02:59.2364888Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-12-01T10:02:59.2399953Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-12-01T10:02:59.2433261Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-12-01T10:02:59.2466325Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-12-01T10:02:59.2503526Z Entering 'third_party/zstd' 2022-12-01T10:02:59.2548115Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2022-12-01T10:02:59.2798918Z Entering 'android/libs/fbjni' 2022-12-01T10:02:59.2832280Z Entering 'third_party/FP16' 2022-12-01T10:02:59.2864896Z Entering 'third_party/FXdiv' 2022-12-01T10:02:59.2902427Z Entering 'third_party/NNPACK' 2022-12-01T10:02:59.2934739Z Entering 'third_party/QNNPACK' 2022-12-01T10:02:59.2970873Z Entering 'third_party/VulkanMemoryAllocator' 2022-12-01T10:02:59.3005623Z Entering 'third_party/XNNPACK' 2022-12-01T10:02:59.3048207Z Entering 'third_party/benchmark' 2022-12-01T10:02:59.3082485Z Entering 'third_party/cpuinfo' 2022-12-01T10:02:59.3115947Z Entering 'third_party/cub' 2022-12-01T10:02:59.3182391Z Entering 'third_party/cudnn_frontend' 2022-12-01T10:02:59.3189398Z Entering 'third_party/cutlass' 2022-12-01T10:02:59.3229682Z Entering 'third_party/eigen' 2022-12-01T10:02:59.3270994Z Entering 'third_party/fbgemm' 2022-12-01T10:02:59.3306258Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-12-01T10:02:59.3339755Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-12-01T10:02:59.3373443Z Entering 'third_party/fbgemm/third_party/googletest' 2022-12-01T10:02:59.3407887Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-12-01T10:02:59.3443099Z Entering 'third_party/flatbuffers' 2022-12-01T10:02:59.3480038Z Entering 'third_party/fmt' 2022-12-01T10:02:59.3513432Z Entering 'third_party/foxi' 2022-12-01T10:02:59.3547473Z Entering 'third_party/gemmlowp/gemmlowp' 2022-12-01T10:02:59.3581420Z Entering 'third_party/gloo' 2022-12-01T10:02:59.3615601Z Entering 'third_party/googletest' 2022-12-01T10:02:59.3650109Z Entering 'third_party/ideep' 2022-12-01T10:02:59.3683582Z Entering 'third_party/ideep/mkl-dnn' 2022-12-01T10:02:59.3718491Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-12-01T10:02:59.3759389Z Entering 'third_party/ios-cmake' 2022-12-01T10:02:59.3792279Z Entering 'third_party/ittapi' 2022-12-01T10:02:59.3826414Z Entering 'third_party/kineto' 2022-12-01T10:02:59.3861916Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-12-01T10:02:59.3896292Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-12-01T10:02:59.3930973Z Entering 'third_party/nccl/nccl' 2022-12-01T10:02:59.3965264Z Entering 'third_party/neon2sse' 2022-12-01T10:02:59.3999604Z Entering 'third_party/nlohmann' 2022-12-01T10:02:59.4034278Z Entering 'third_party/onnx' 2022-12-01T10:02:59.4078864Z Entering 'third_party/onnx/third_party/benchmark' 2022-12-01T10:02:59.4112730Z Entering 'third_party/onnx/third_party/pybind11' 2022-12-01T10:02:59.4149364Z Entering 'third_party/onnx-tensorrt' 2022-12-01T10:02:59.4183761Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-12-01T10:02:59.4220970Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-12-01T10:02:59.4256103Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-12-01T10:02:59.4290312Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-12-01T10:02:59.4330017Z Entering 'third_party/pocketfft' 2022-12-01T10:02:59.4365076Z Entering 'third_party/protobuf' 2022-12-01T10:02:59.4405126Z Entering 'third_party/protobuf/third_party/benchmark' 2022-12-01T10:02:59.4439894Z Entering 'third_party/protobuf/third_party/googletest' 2022-12-01T10:02:59.4474213Z Entering 'third_party/psimd' 2022-12-01T10:02:59.4509217Z Entering 'third_party/pthreadpool' 2022-12-01T10:02:59.4543442Z Entering 'third_party/pybind11' 2022-12-01T10:02:59.4578081Z Entering 'third_party/python-enum' 2022-12-01T10:02:59.4611686Z Entering 'third_party/python-peachpy' 2022-12-01T10:02:59.4645040Z Entering 'third_party/python-six' 2022-12-01T10:02:59.4680224Z Entering 'third_party/sleef' 2022-12-01T10:02:59.4713525Z Entering 'third_party/tbb' 2022-12-01T10:02:59.4750054Z Entering 'third_party/tensorpipe' 2022-12-01T10:02:59.4784733Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-12-01T10:02:59.4818929Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-12-01T10:02:59.4852034Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-12-01T10:02:59.4885662Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-12-01T10:02:59.4920146Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-12-01T10:02:59.4956318Z Entering 'third_party/zstd' 2022-12-01T10:02:59.4998158Z ##[endgroup] 2022-12-01T10:02:59.5036748Z [command]/usr/bin/git log -1 --format='%H' 2022-12-01T10:02:59.5063115Z 'c13d400bffe90e16b96520bbc8a41a6f0c9cd584' 2022-12-01T10:02:59.5192134Z Prepare all required actions 2022-12-01T10:02:59.5217643Z ##[group]Run ./.github/actions/setup-linux 2022-12-01T10:02:59.5217847Z env: 2022-12-01T10:02:59.5218023Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:02:59.5218196Z ##[endgroup] 2022-12-01T10:02:59.5297745Z ##[group]Run set -euo pipefail 2022-12-01T10:02:59.5297977Z set -euo pipefail 2022-12-01T10:02:59.5298187Z function get_ec2_metadata() { 2022-12-01T10:02:59.5298422Z  # Pulled from instance metadata endpoint for EC2 2022-12-01T10:02:59.5298779Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2022-12-01T10:02:59.5299080Z  category=$1 2022-12-01T10:02:59.5299385Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2022-12-01T10:02:59.5299613Z } 2022-12-01T10:02:59.5299914Z echo "ami-id: $(get_ec2_metadata ami-id)" 2022-12-01T10:02:59.5300186Z echo "instance-id: $(get_ec2_metadata instance-id)" 2022-12-01T10:02:59.5300465Z echo "instance-type: $(get_ec2_metadata instance-type)" 2022-12-01T10:02:59.5300715Z echo "system info $(uname -a)" 2022-12-01T10:02:59.5311973Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:02:59.5312191Z env: 2022-12-01T10:02:59.5312366Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:02:59.5312554Z ##[endgroup] 2022-12-01T10:02:59.5432401Z ami-id: ami-096198a0bccc6bad4 2022-12-01T10:02:59.5481765Z instance-id: i-092913c5169eb9727 2022-12-01T10:02:59.5527777Z instance-type: c5.2xlarge 2022-12-01T10:02:59.5534462Z system info Linux ip-10-0-1-92.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 2022-12-01T10:02:59.5550811Z ##[group]Run if systemctl is-active --quiet docker; then 2022-12-01T10:02:59.5551095Z if systemctl is-active --quiet docker; then 2022-12-01T10:02:59.5551336Z  echo "Docker daemon is running..."; 2022-12-01T10:02:59.5551548Z else 2022-12-01T10:02:59.5551786Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2022-12-01T10:02:59.5552014Z fi 2022-12-01T10:02:59.5563164Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:02:59.5563382Z env: 2022-12-01T10:02:59.5563559Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:02:59.5563735Z ##[endgroup] 2022-12-01T10:02:59.5622780Z Docker daemon is running... 2022-12-01T10:02:59.5641557Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-12-01T10:02:59.5641910Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-12-01T10:02:59.5642179Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-12-01T10:02:59.5642555Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \ 2022-12-01T10:02:59.5642910Z  --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2022-12-01T10:02:59.5652990Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:02:59.5653192Z env: 2022-12-01T10:02:59.5653372Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:02:59.5653569Z AWS_RETRY_MODE: standard 2022-12-01T10:02:59.5653742Z AWS_MAX_ATTEMPTS: 5 2022-12-01T10:02:59.5653942Z AWS_DEFAULT_REGION: us-east-1 2022-12-01T10:02:59.5654131Z ##[endgroup] 2022-12-01T10:03:01.5384150Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2022-12-01T10:03:01.5384501Z Configure a credential helper to remove this warning. See 2022-12-01T10:03:01.5385359Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2022-12-01T10:03:01.5385558Z 2022-12-01T10:03:01.5385634Z Login Succeeded 2022-12-01T10:03:01.5419832Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-12-01T10:03:01.5420188Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-12-01T10:03:01.5420596Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-12-01T10:03:01.5431782Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:03:01.5432031Z env: 2022-12-01T10:03:01.5432238Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:03:01.5432443Z ##[endgroup] 2022-12-01T10:03:01.5525088Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2022-12-01T10:03:01.5525324Z with: 2022-12-01T10:03:01.5525676Z github-secret: *** 2022-12-01T10:03:01.5525879Z activate-with-label: false 2022-12-01T10:03:01.5526058Z label: with-ssh 2022-12-01T10:03:01.5526249Z remove-existing-keys: true 2022-12-01T10:03:01.5526429Z env: 2022-12-01T10:03:01.5526585Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:03:01.5526769Z ##[endgroup] 2022-12-01T10:03:01.9977276Z Grabbing public ssh keys from https://github.com/charlie-wt.keys 2022-12-01T10:03:02.0892041Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2022-12-01T10:03:02.0905245Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2022-12-01T10:03:02.0930483Z Login using: ssh ec2-user@ec2-3-233-240-93.compute-1.amazonaws.com 2022-12-01T10:03:02.0993341Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2022-12-01T10:03:02.0993594Z with: 2022-12-01T10:03:02.0993912Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b 2022-12-01T10:03:02.0994229Z env: 2022-12-01T10:03:02.0994400Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:03:02.0994573Z ##[endgroup] 2022-12-01T10:03:02.1007551Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-12-01T10:03:02.1007930Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-12-01T10:03:02.1008198Z # ignore output since only exit code is used for conditional 2022-12-01T10:03:02.1008482Z # only pull docker image if it's not available locally 2022-12-01T10:03:02.1008772Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2022-12-01T10:03:02.1009072Z  retry docker pull "${DOCKER_IMAGE}" 2022-12-01T10:03:02.1009268Z fi 2022-12-01T10:03:02.1020493Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:03:02.1020712Z env: 2022-12-01T10:03:02.1020882Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:03:02.1021220Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b 2022-12-01T10:03:02.1021542Z ##[endgroup] 2022-12-01T10:03:02.3665137Z fa72f5a0a230eb632055220542038bd4ceca184b: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7 2022-12-01T10:03:02.3707482Z 675920708c8b: Pulling fs layer 2022-12-01T10:03:02.3708252Z 3a31ffa0162f: Pulling fs layer 2022-12-01T10:03:02.3708620Z fe9b24415086: Pulling fs layer 2022-12-01T10:03:02.3708958Z 422bea31bd2a: Pulling fs layer 2022-12-01T10:03:02.3709292Z 27ad548906a1: Pulling fs layer 2022-12-01T10:03:02.3709603Z efe762c046e3: Pulling fs layer 2022-12-01T10:03:02.3709905Z a12af00c9224: Pulling fs layer 2022-12-01T10:03:02.3712202Z 49809737e89d: Pulling fs layer 2022-12-01T10:03:02.3712531Z 4bd15b741963: Pulling fs layer 2022-12-01T10:03:02.3712786Z 9fdf20fee962: Pulling fs layer 2022-12-01T10:03:02.3712982Z 27ad548906a1: Waiting 2022-12-01T10:03:02.3713169Z efe762c046e3: Waiting 2022-12-01T10:03:02.3713406Z 85d2fa9daf54: Pulling fs layer 2022-12-01T10:03:02.3714680Z a12af00c9224: Waiting 2022-12-01T10:03:02.3715009Z e3608d8edfea: Pulling fs layer 2022-12-01T10:03:02.3715314Z 4bd15b741963: Waiting 2022-12-01T10:03:02.3715583Z 422bea31bd2a: Waiting 2022-12-01T10:03:02.3715741Z 49809737e89d: Waiting 2022-12-01T10:03:02.3715910Z 9fdf20fee962: Waiting 2022-12-01T10:03:02.3716096Z 63ae5eb88541: Pulling fs layer 2022-12-01T10:03:02.3716277Z 2b6269e5f012: Pulling fs layer 2022-12-01T10:03:02.3716461Z e3608d8edfea: Waiting 2022-12-01T10:03:02.3716653Z 49405b81ba46: Pulling fs layer 2022-12-01T10:03:02.3716822Z 63ae5eb88541: Waiting 2022-12-01T10:03:02.3717003Z f5bd51a8525a: Pulling fs layer 2022-12-01T10:03:02.3717183Z 2b6269e5f012: Waiting 2022-12-01T10:03:02.3717357Z dce0658383e3: Pulling fs layer 2022-12-01T10:03:02.3717553Z fda59f894d4a: Pulling fs layer 2022-12-01T10:03:02.3723062Z f5bd51a8525a: Waiting 2022-12-01T10:03:02.3724759Z dce0658383e3: Waiting 2022-12-01T10:03:02.3725157Z c4cd62aabb9c: Pulling fs layer 2022-12-01T10:03:02.3725524Z d84da0b3b529: Pulling fs layer 2022-12-01T10:03:02.3725852Z c4cd62aabb9c: Waiting 2022-12-01T10:03:02.3726180Z 81e2ac298c61: Pulling fs layer 2022-12-01T10:03:02.3726477Z 38c05cfe4070: Pulling fs layer 2022-12-01T10:03:02.3726834Z af4d19351fdf: Pulling fs layer 2022-12-01T10:03:02.3727207Z d1d5291b2ab5: Pulling fs layer 2022-12-01T10:03:02.3727483Z 17be6d40a7e6: Pulling fs layer 2022-12-01T10:03:02.3727704Z 7334631630ca: Pulling fs layer 2022-12-01T10:03:02.3728012Z b7010c1141c8: Pulling fs layer 2022-12-01T10:03:02.3728509Z d1d5291b2ab5: Waiting 2022-12-01T10:03:02.3728829Z 1d69ebcc1824: Pulling fs layer 2022-12-01T10:03:02.3729159Z 17be6d40a7e6: Waiting 2022-12-01T10:03:02.3729464Z 7334631630ca: Waiting 2022-12-01T10:03:02.3729865Z bbd83461fb41: Pulling fs layer 2022-12-01T10:03:02.3730214Z bcd5197bc3b1: Pulling fs layer 2022-12-01T10:03:02.3730517Z b80f27646ab6: Pulling fs layer 2022-12-01T10:03:02.3730819Z 1d69ebcc1824: Waiting 2022-12-01T10:03:02.3731152Z 866c2acc4910: Pulling fs layer 2022-12-01T10:03:02.3731502Z e39e0e87bf58: Pulling fs layer 2022-12-01T10:03:02.3731837Z bbd83461fb41: Waiting 2022-12-01T10:03:02.3732150Z 31e492142d52: Pulling fs layer 2022-12-01T10:03:02.3732480Z bcd5197bc3b1: Waiting 2022-12-01T10:03:02.3732791Z 7d95e9c9554c: Pulling fs layer 2022-12-01T10:03:02.3733063Z b80f27646ab6: Waiting 2022-12-01T10:03:02.3733345Z f405cc5c9c77: Pulling fs layer 2022-12-01T10:03:02.3733633Z e39e0e87bf58: Waiting 2022-12-01T10:03:02.3733919Z a1d89816b283: Pulling fs layer 2022-12-01T10:03:02.3734284Z 38d7b060d3d4: Pulling fs layer 2022-12-01T10:03:02.3734612Z f405cc5c9c77: Waiting 2022-12-01T10:03:02.3734781Z 7d95e9c9554c: Waiting 2022-12-01T10:03:02.3734961Z 0b48564513b3: Pulling fs layer 2022-12-01T10:03:02.3735139Z a1d89816b283: Waiting 2022-12-01T10:03:02.3735326Z c517839d268c: Pulling fs layer 2022-12-01T10:03:02.3735510Z 9200027c6378: Pulling fs layer 2022-12-01T10:03:02.3735687Z 0b48564513b3: Waiting 2022-12-01T10:03:02.3735852Z 1a68c742e877: Pulling fs layer 2022-12-01T10:03:02.3736030Z c517839d268c: Waiting 2022-12-01T10:03:02.3736193Z 9200027c6378: Waiting 2022-12-01T10:03:02.3736357Z 085aceb64709: Pulling fs layer 2022-12-01T10:03:02.3736533Z 1a68c742e877: Waiting 2022-12-01T10:03:02.3736713Z 9fad26f5e1f4: Pulling fs layer 2022-12-01T10:03:02.3736902Z 3454ac6b1f68: Pulling fs layer 2022-12-01T10:03:02.3737078Z 38599e0ca649: Pulling fs layer 2022-12-01T10:03:02.3737265Z a16f0b09d742: Pulling fs layer 2022-12-01T10:03:02.3737455Z 100cd65220ba: Pulling fs layer 2022-12-01T10:03:02.3737634Z 9d7685da3dfa: Pulling fs layer 2022-12-01T10:03:02.3737811Z 38599e0ca649: Waiting 2022-12-01T10:03:02.3737989Z e69408c62fb4: Pulling fs layer 2022-12-01T10:03:02.3738164Z 1530f151c058: Pulling fs layer 2022-12-01T10:03:02.3738342Z a16f0b09d742: Waiting 2022-12-01T10:03:02.3738523Z 3aeaec05c672: Pulling fs layer 2022-12-01T10:03:02.3738699Z 99763bc3448a: Pulling fs layer 2022-12-01T10:03:02.3738874Z 1530f151c058: Waiting 2022-12-01T10:03:02.3739054Z cec809ad8b3c: Pulling fs layer 2022-12-01T10:03:02.3739229Z 3aeaec05c672: Waiting 2022-12-01T10:03:02.3739487Z 677fdb68d2c6: Pulling fs layer 2022-12-01T10:03:02.3739676Z 92d94c0f43b4: Pulling fs layer 2022-12-01T10:03:02.3739854Z a90bc9ac3f47: Pulling fs layer 2022-12-01T10:03:02.3740031Z 99763bc3448a: Waiting 2022-12-01T10:03:02.3740212Z 8ef4cf23833b: Pulling fs layer 2022-12-01T10:03:02.3740384Z 677fdb68d2c6: Waiting 2022-12-01T10:03:02.3740555Z cec809ad8b3c: Waiting 2022-12-01T10:03:02.3740726Z 8ef4cf23833b: Waiting 2022-12-01T10:03:02.3740887Z a90bc9ac3f47: Waiting 2022-12-01T10:03:02.3741057Z 92d94c0f43b4: Waiting 2022-12-01T10:03:02.4618550Z 3a31ffa0162f: Download complete 2022-12-01T10:03:02.5532886Z 422bea31bd2a: Verifying Checksum 2022-12-01T10:03:02.5533265Z 422bea31bd2a: Download complete 2022-12-01T10:03:02.6307001Z 27ad548906a1: Verifying Checksum 2022-12-01T10:03:02.6307395Z 27ad548906a1: Download complete 2022-12-01T10:03:02.7291868Z 675920708c8b: Download complete 2022-12-01T10:03:02.7603155Z efe762c046e3: Verifying Checksum 2022-12-01T10:03:02.8129724Z a12af00c9224: Download complete 2022-12-01T10:03:02.8370917Z 49809737e89d: Download complete 2022-12-01T10:03:02.8901363Z 4bd15b741963: Verifying Checksum 2022-12-01T10:03:02.8901761Z 4bd15b741963: Download complete 2022-12-01T10:03:02.9283745Z 9fdf20fee962: Verifying Checksum 2022-12-01T10:03:02.9284166Z 9fdf20fee962: Download complete 2022-12-01T10:03:02.9964987Z e3608d8edfea: Download complete 2022-12-01T10:03:03.0848876Z 63ae5eb88541: Verifying Checksum 2022-12-01T10:03:03.0849278Z 63ae5eb88541: Download complete 2022-12-01T10:03:03.4444498Z 675920708c8b: Pull complete 2022-12-01T10:03:03.5566237Z 3a31ffa0162f: Pull complete 2022-12-01T10:03:03.7696956Z 85d2fa9daf54: Verifying Checksum 2022-12-01T10:03:03.7697540Z 85d2fa9daf54: Download complete 2022-12-01T10:03:03.8443400Z 49405b81ba46: Verifying Checksum 2022-12-01T10:03:03.8443846Z 49405b81ba46: Download complete 2022-12-01T10:03:03.9425760Z f5bd51a8525a: Verifying Checksum 2022-12-01T10:03:03.9426068Z f5bd51a8525a: Download complete 2022-12-01T10:03:04.4440828Z dce0658383e3: Verifying Checksum 2022-12-01T10:03:04.4441292Z dce0658383e3: Download complete 2022-12-01T10:03:04.5238097Z fda59f894d4a: Verifying Checksum 2022-12-01T10:03:04.5238597Z fda59f894d4a: Download complete 2022-12-01T10:03:04.6230722Z c4cd62aabb9c: Verifying Checksum 2022-12-01T10:03:04.6231213Z c4cd62aabb9c: Download complete 2022-12-01T10:03:04.6880010Z d84da0b3b529: Download complete 2022-12-01T10:03:04.7692123Z 81e2ac298c61: Verifying Checksum 2022-12-01T10:03:04.7692378Z 81e2ac298c61: Download complete 2022-12-01T10:03:04.8499023Z 38c05cfe4070: Verifying Checksum 2022-12-01T10:03:04.8499516Z 38c05cfe4070: Download complete 2022-12-01T10:03:04.9254462Z af4d19351fdf: Verifying Checksum 2022-12-01T10:03:04.9254734Z af4d19351fdf: Download complete 2022-12-01T10:03:05.0043927Z d1d5291b2ab5: Verifying Checksum 2022-12-01T10:03:05.0044214Z d1d5291b2ab5: Download complete 2022-12-01T10:03:05.5027845Z fe9b24415086: Verifying Checksum 2022-12-01T10:03:05.5028156Z fe9b24415086: Download complete 2022-12-01T10:03:05.5916149Z 7334631630ca: Download complete 2022-12-01T10:03:05.7016191Z b7010c1141c8: Download complete 2022-12-01T10:03:05.7840823Z 1d69ebcc1824: Verifying Checksum 2022-12-01T10:03:05.7841268Z 1d69ebcc1824: Download complete 2022-12-01T10:03:05.8612501Z bbd83461fb41: Verifying Checksum 2022-12-01T10:03:05.8612763Z bbd83461fb41: Download complete 2022-12-01T10:03:05.9473431Z bcd5197bc3b1: Verifying Checksum 2022-12-01T10:03:05.9473700Z bcd5197bc3b1: Download complete 2022-12-01T10:03:05.9569108Z 17be6d40a7e6: Verifying Checksum 2022-12-01T10:03:05.9569542Z 17be6d40a7e6: Download complete 2022-12-01T10:03:06.0413334Z 866c2acc4910: Verifying Checksum 2022-12-01T10:03:06.0413764Z 866c2acc4910: Download complete 2022-12-01T10:03:06.1224409Z e39e0e87bf58: Verifying Checksum 2022-12-01T10:03:06.1224692Z e39e0e87bf58: Download complete 2022-12-01T10:03:06.2438701Z 31e492142d52: Verifying Checksum 2022-12-01T10:03:06.2439066Z 31e492142d52: Download complete 2022-12-01T10:03:06.3276170Z 7d95e9c9554c: Verifying Checksum 2022-12-01T10:03:06.3276572Z 7d95e9c9554c: Download complete 2022-12-01T10:03:06.4018686Z f405cc5c9c77: Verifying Checksum 2022-12-01T10:03:06.4019171Z f405cc5c9c77: Download complete 2022-12-01T10:03:06.5118513Z a1d89816b283: Verifying Checksum 2022-12-01T10:03:06.5119007Z a1d89816b283: Download complete 2022-12-01T10:03:06.6056008Z 38d7b060d3d4: Download complete 2022-12-01T10:03:06.7106969Z 0b48564513b3: Verifying Checksum 2022-12-01T10:03:06.7107430Z 0b48564513b3: Download complete 2022-12-01T10:03:06.8085322Z c517839d268c: Verifying Checksum 2022-12-01T10:03:06.8085758Z c517839d268c: Download complete 2022-12-01T10:03:06.8981620Z 9200027c6378: Download complete 2022-12-01T10:03:07.0989169Z 1a68c742e877: Download complete 2022-12-01T10:03:07.1825793Z 085aceb64709: Verifying Checksum 2022-12-01T10:03:07.1826189Z 085aceb64709: Download complete 2022-12-01T10:03:07.6545174Z 9fad26f5e1f4: Verifying Checksum 2022-12-01T10:03:07.6547729Z 9fad26f5e1f4: Download complete 2022-12-01T10:03:07.7295224Z 3454ac6b1f68: Verifying Checksum 2022-12-01T10:03:07.7295463Z 3454ac6b1f68: Download complete 2022-12-01T10:03:07.8086575Z 38599e0ca649: Download complete 2022-12-01T10:03:07.8799591Z a16f0b09d742: Verifying Checksum 2022-12-01T10:03:07.8800276Z a16f0b09d742: Download complete 2022-12-01T10:03:07.9659723Z 100cd65220ba: Verifying Checksum 2022-12-01T10:03:07.9660137Z 100cd65220ba: Download complete 2022-12-01T10:03:08.1864474Z b80f27646ab6: Verifying Checksum 2022-12-01T10:03:08.1864996Z b80f27646ab6: Download complete 2022-12-01T10:03:08.2101607Z 9d7685da3dfa: Verifying Checksum 2022-12-01T10:03:08.2102023Z 9d7685da3dfa: Download complete 2022-12-01T10:03:08.2826789Z e69408c62fb4: Verifying Checksum 2022-12-01T10:03:08.2827394Z e69408c62fb4: Download complete 2022-12-01T10:03:08.2917181Z 1530f151c058: Verifying Checksum 2022-12-01T10:03:08.2917542Z 1530f151c058: Download complete 2022-12-01T10:03:08.3735542Z 99763bc3448a: Download complete 2022-12-01T10:03:08.4602611Z cec809ad8b3c: Verifying Checksum 2022-12-01T10:03:08.4603027Z cec809ad8b3c: Download complete 2022-12-01T10:03:08.5232534Z 677fdb68d2c6: Download complete 2022-12-01T10:03:08.6035920Z 92d94c0f43b4: Download complete 2022-12-01T10:03:08.6894811Z a90bc9ac3f47: Verifying Checksum 2022-12-01T10:03:08.6895518Z a90bc9ac3f47: Download complete 2022-12-01T10:03:09.2810961Z 8ef4cf23833b: Verifying Checksum 2022-12-01T10:03:09.2811327Z 8ef4cf23833b: Download complete 2022-12-01T10:03:11.5194243Z 3aeaec05c672: Verifying Checksum 2022-12-01T10:03:11.5194921Z 3aeaec05c672: Download complete 2022-12-01T10:03:12.0721921Z fe9b24415086: Pull complete 2022-12-01T10:03:12.2320011Z 422bea31bd2a: Pull complete 2022-12-01T10:03:12.4014574Z 27ad548906a1: Pull complete 2022-12-01T10:03:12.5585687Z efe762c046e3: Pull complete 2022-12-01T10:03:12.7779377Z a12af00c9224: Pull complete 2022-12-01T10:03:13.0336368Z 49809737e89d: Pull complete 2022-12-01T10:03:13.2608399Z 4bd15b741963: Pull complete 2022-12-01T10:03:13.4839569Z 9fdf20fee962: Pull complete 2022-12-01T10:03:15.0407939Z 2b6269e5f012: Verifying Checksum 2022-12-01T10:03:15.0408372Z 2b6269e5f012: Download complete 2022-12-01T10:03:15.3687231Z 85d2fa9daf54: Pull complete 2022-12-01T10:03:15.5896839Z e3608d8edfea: Pull complete 2022-12-01T10:03:15.7324430Z 63ae5eb88541: Pull complete 2022-12-01T10:03:36.1293216Z 2b6269e5f012: Pull complete 2022-12-01T10:03:36.4993432Z 49405b81ba46: Pull complete 2022-12-01T10:03:36.7512540Z f5bd51a8525a: Pull complete 2022-12-01T10:03:37.5708623Z dce0658383e3: Pull complete 2022-12-01T10:03:37.6978188Z fda59f894d4a: Pull complete 2022-12-01T10:03:37.8698696Z c4cd62aabb9c: Pull complete 2022-12-01T10:03:38.0326788Z d84da0b3b529: Pull complete 2022-12-01T10:03:38.1430779Z 81e2ac298c61: Pull complete 2022-12-01T10:03:38.2744392Z 38c05cfe4070: Pull complete 2022-12-01T10:03:38.4131846Z af4d19351fdf: Pull complete 2022-12-01T10:03:38.5451445Z d1d5291b2ab5: Pull complete 2022-12-01T10:03:40.3591931Z 17be6d40a7e6: Pull complete 2022-12-01T10:03:40.5032601Z 7334631630ca: Pull complete 2022-12-01T10:03:40.6936491Z b7010c1141c8: Pull complete 2022-12-01T10:03:40.8908071Z 1d69ebcc1824: Pull complete 2022-12-01T10:03:41.1167081Z bbd83461fb41: Pull complete 2022-12-01T10:03:41.3464366Z bcd5197bc3b1: Pull complete 2022-12-01T10:03:45.7927339Z b80f27646ab6: Pull complete 2022-12-01T10:03:46.0258793Z 866c2acc4910: Pull complete 2022-12-01T10:03:46.2704172Z e39e0e87bf58: Pull complete 2022-12-01T10:03:46.4275617Z 31e492142d52: Pull complete 2022-12-01T10:03:46.6414370Z 7d95e9c9554c: Pull complete 2022-12-01T10:03:46.8742340Z f405cc5c9c77: Pull complete 2022-12-01T10:03:46.9735526Z a1d89816b283: Pull complete 2022-12-01T10:03:47.0650457Z 38d7b060d3d4: Pull complete 2022-12-01T10:03:47.1445072Z 0b48564513b3: Pull complete 2022-12-01T10:03:47.2418012Z c517839d268c: Pull complete 2022-12-01T10:03:47.3285343Z 9200027c6378: Pull complete 2022-12-01T10:03:47.4195274Z 1a68c742e877: Pull complete 2022-12-01T10:03:47.5179466Z 085aceb64709: Pull complete 2022-12-01T10:03:48.6120501Z 9fad26f5e1f4: Pull complete 2022-12-01T10:03:48.7036658Z 3454ac6b1f68: Pull complete 2022-12-01T10:03:48.7999748Z 38599e0ca649: Pull complete 2022-12-01T10:03:48.9038520Z a16f0b09d742: Pull complete 2022-12-01T10:03:48.9832479Z 100cd65220ba: Pull complete 2022-12-01T10:03:49.2863897Z 9d7685da3dfa: Pull complete 2022-12-01T10:03:49.3829726Z e69408c62fb4: Pull complete 2022-12-01T10:03:49.4669491Z 1530f151c058: Pull complete 2022-12-01T10:03:53.3628644Z 3aeaec05c672: Pull complete 2022-12-01T10:03:53.6115427Z 99763bc3448a: Pull complete 2022-12-01T10:03:53.6969354Z cec809ad8b3c: Pull complete 2022-12-01T10:03:53.7983449Z 677fdb68d2c6: Pull complete 2022-12-01T10:03:53.9000374Z 92d94c0f43b4: Pull complete 2022-12-01T10:03:53.9787203Z a90bc9ac3f47: Pull complete 2022-12-01T10:03:55.4935416Z 8ef4cf23833b: Pull complete 2022-12-01T10:03:55.5885530Z Digest: sha256:7d4daa1530ad815417dd92428a12643a4a8b0e93e820e2eb2d4e8818a4e57ba2 2022-12-01T10:03:55.6283549Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b 2022-12-01T10:03:55.6523390Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b 2022-12-01T10:03:55.6583380Z ##[group]Run python3 -m pip install psutil==5.9.1 2022-12-01T10:03:55.6583682Z python3 -m pip install psutil==5.9.1 2022-12-01T10:03:55.6584017Z python3 -m pip install pynvml==11.4.1 2022-12-01T10:03:55.6584366Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2022-12-01T10:03:55.6584703Z echo "::set-output name=monitor-script-pid::${!}" 2022-12-01T10:03:55.6595673Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:03:55.6596067Z env: 2022-12-01T10:03:55.6596309Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:03:55.6596568Z ##[endgroup] 2022-12-01T10:03:55.8685839Z Defaulting to user installation because normal site-packages is not writeable 2022-12-01T10:03:55.8872504Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2022-12-01T10:03:56.4026439Z Defaulting to user installation because normal site-packages is not writeable 2022-12-01T10:03:56.4213516Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1) 2022-12-01T10:03:56.7290951Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-12-01T10:03:56.7332054Z Prepare all required actions 2022-12-01T10:03:56.7332305Z Getting action download info 2022-12-01T10:03:56.9047426Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea) 2022-12-01T10:03:57.0651599Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869) 2022-12-01T10:03:57.2003605Z ##[group]Run ./.github/actions/download-build-artifacts 2022-12-01T10:03:57.2003834Z with: 2022-12-01T10:03:57.2004011Z name: linux-focal-py3.7-gcc7 2022-12-01T10:03:57.2004202Z env: 2022-12-01T10:03:57.2004377Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:03:57.2004552Z ##[endgroup] 2022-12-01T10:03:57.2114236Z ##[group]Run seemethere/download-artifact-s3@v4 2022-12-01T10:03:57.2114444Z with: 2022-12-01T10:03:57.2114637Z name: linux-focal-py3.7-gcc7 2022-12-01T10:03:57.2114858Z s3-bucket: gha-artifacts 2022-12-01T10:03:57.2115056Z region: us-east-1 2022-12-01T10:03:57.2115213Z env: 2022-12-01T10:03:57.2115422Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:03:57.2115595Z ##[endgroup] 2022-12-01T10:03:57.6339262Z Found 1 objects with prefix pytorch/pytorch/3591403534/linux-focal-py3.7-gcc7/ 2022-12-01T10:03:57.6339927Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-12-01T10:04:00.6627461Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-12-01T10:04:00.6627912Z 2022-12-01T10:04:00.6630481Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-12-01T10:04:00.6632007Z Artifact download has finished successfully 2022-12-01T10:04:00.6748237Z ##[group]Run unzip -o artifacts.zip 2022-12-01T10:04:00.6748544Z unzip -o artifacts.zip 2022-12-01T10:04:00.6759666Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:04:00.6759884Z env: 2022-12-01T10:04:00.6760059Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:04:00.6760232Z ##[endgroup] 2022-12-01T10:04:00.6825823Z Archive: artifacts.zip 2022-12-01T10:04:00.6827099Z creating: dist/ 2022-12-01T10:04:01.4397288Z inflating: dist/torch-1.13.0a0+gitc13d400-cp37-cp37m-linux_x86_64.whl 2022-12-01T10:04:01.4397590Z creating: build/custom_test_artifacts/ 2022-12-01T10:04:01.4397909Z creating: build/custom_test_artifacts/custom-op-build/ 2022-12-01T10:04:01.4398457Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-12-01T10:04:01.4400649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-12-01T10:04:01.4401165Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2022-12-01T10:04:01.4401667Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-12-01T10:04:01.4402116Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-12-01T10:04:01.4402520Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-12-01T10:04:01.4404377Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-12-01T10:04:01.4405905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-12-01T10:04:01.4406330Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-12-01T10:04:01.4406747Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-12-01T10:04:01.4408680Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-12-01T10:04:01.4410060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-12-01T10:04:01.4411856Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-12-01T10:04:01.4412336Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-12-01T10:04:01.4413761Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-12-01T10:04:01.4414483Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-12-01T10:04:01.4414899Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-12-01T10:04:01.4415444Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-12-01T10:04:01.4415908Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-12-01T10:04:01.4416316Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-12-01T10:04:01.4416764Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2022-12-01T10:04:01.4417231Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2022-12-01T10:04:01.4417831Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-12-01T10:04:01.4418263Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-12-01T10:04:01.4419327Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-12-01T10:04:01.4420090Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-12-01T10:04:01.4420594Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-12-01T10:04:01.4421651Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-12-01T10:04:01.4422701Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-12-01T10:04:01.4437560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2022-12-01T10:04:01.4527430Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-12-01T10:04:01.4528003Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-12-01T10:04:01.4528467Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2022-12-01T10:04:01.4529107Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2022-12-01T10:04:01.4529694Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-12-01T10:04:01.4530497Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-12-01T10:04:01.4531397Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-12-01T10:04:01.4532025Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-12-01T10:04:01.4532469Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-12-01T10:04:01.4532927Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-12-01T10:04:01.4533378Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-12-01T10:04:01.4546172Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2022-12-01T10:04:01.4609154Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-12-01T10:04:01.4609983Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-12-01T10:04:01.4610723Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-12-01T10:04:01.4611171Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-12-01T10:04:01.4611576Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-12-01T10:04:01.4611977Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-12-01T10:04:01.4612639Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-12-01T10:04:01.4613344Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-12-01T10:04:01.4614036Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-12-01T10:04:01.4684942Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-12-01T10:04:01.4732930Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-12-01T10:04:01.4733311Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-12-01T10:04:01.4733721Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-12-01T10:04:01.4736393Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-12-01T10:04:01.4736844Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2022-12-01T10:04:01.4737402Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-12-01T10:04:01.4737811Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-12-01T10:04:01.4738308Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-12-01T10:04:01.4739510Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-12-01T10:04:01.4740961Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-12-01T10:04:01.4741542Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-12-01T10:04:01.4741963Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-12-01T10:04:01.4743451Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-12-01T10:04:01.4744809Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-12-01T10:04:01.4746298Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-12-01T10:04:01.4746850Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-12-01T10:04:01.4748108Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-12-01T10:04:01.4748968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-12-01T10:04:01.4749524Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-12-01T10:04:01.4750080Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-12-01T10:04:01.4750625Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-12-01T10:04:01.4751148Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-12-01T10:04:01.4751757Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2022-12-01T10:04:01.4752406Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2022-12-01T10:04:01.4752915Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-12-01T10:04:01.4753560Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-12-01T10:04:01.4754184Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-12-01T10:04:01.4754795Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-12-01T10:04:01.4755255Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-12-01T10:04:01.4755698Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-12-01T10:04:01.4756127Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-12-01T10:04:01.4770344Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2022-12-01T10:04:01.4818771Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-12-01T10:04:01.4819491Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-12-01T10:04:01.4820068Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-12-01T10:04:01.4820596Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-12-01T10:04:01.4821014Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-12-01T10:04:01.4821522Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-12-01T10:04:01.4822729Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-12-01T10:04:01.4823417Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-12-01T10:04:01.4823919Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-12-01T10:04:01.4861762Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-12-01T10:04:01.4862220Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-12-01T10:04:01.4862573Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-12-01T10:04:01.4864814Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-12-01T10:04:01.4865290Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2022-12-01T10:04:01.4865792Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-12-01T10:04:01.4866251Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-12-01T10:04:01.4866686Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-12-01T10:04:01.4868141Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-12-01T10:04:01.4869483Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-12-01T10:04:01.4869975Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-12-01T10:04:01.4870419Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-12-01T10:04:01.4872082Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-12-01T10:04:01.4873456Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-12-01T10:04:01.4874781Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-12-01T10:04:01.4875349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-12-01T10:04:01.4876829Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-12-01T10:04:01.4877517Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-12-01T10:04:01.4878000Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-12-01T10:04:01.4878816Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-12-01T10:04:01.4879328Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-12-01T10:04:01.4879874Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-12-01T10:04:01.4880437Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2022-12-01T10:04:01.4881006Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2022-12-01T10:04:01.4881529Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-12-01T10:04:01.4882019Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-12-01T10:04:01.4882525Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-12-01T10:04:01.4883123Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-12-01T10:04:01.4883727Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-12-01T10:04:01.4884269Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-12-01T10:04:01.4884744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-12-01T10:04:01.4887677Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2022-12-01T10:04:01.5000318Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-12-01T10:04:01.5000900Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-12-01T10:04:01.5001569Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2022-12-01T10:04:01.5002127Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2022-12-01T10:04:01.5002701Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-12-01T10:04:01.5003205Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-12-01T10:04:01.5003818Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-12-01T10:04:01.5004367Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-12-01T10:04:01.5004958Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-12-01T10:04:01.5005454Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-12-01T10:04:01.5005938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-12-01T10:04:01.5020825Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2022-12-01T10:04:01.5065325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-12-01T10:04:01.5066053Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-12-01T10:04:01.5066598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-12-01T10:04:01.5067182Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-12-01T10:04:01.5067603Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-12-01T10:04:01.5068223Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-12-01T10:04:01.5069369Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-12-01T10:04:01.5070185Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-12-01T10:04:01.5070714Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-12-01T10:04:01.5161861Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-12-01T10:04:01.5197107Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-12-01T10:04:01.5197397Z creating: build/lib/ 2022-12-01T10:04:01.5198033Z inflating: build/lib/libclog.a 2022-12-01T10:04:01.5249190Z inflating: build/lib/libgtest.a 2022-12-01T10:04:01.5257008Z inflating: build/lib/libpthreadpool.a 2022-12-01T10:04:01.5328217Z inflating: build/lib/libbenchmark.a 2022-12-01T10:04:01.5335088Z inflating: build/lib/libittnotify.a 2022-12-01T10:04:01.5417251Z inflating: build/lib/libprotobuf-lite.a 2022-12-01T10:04:01.5442051Z inflating: build/lib/libtensorpipe_uv.a 2022-12-01T10:04:01.5531913Z inflating: build/lib/libgloo.a 2022-12-01T10:04:01.5591719Z inflating: build/lib/libasmjit.a 2022-12-01T10:04:01.6005782Z inflating: build/lib/libprotobuf.a 2022-12-01T10:04:01.6020762Z inflating: build/lib/libfmt.a 2022-12-01T10:04:01.6021493Z inflating: build/lib/libfoxi_loader.a 2022-12-01T10:04:01.6081353Z inflating: build/lib/libc10.so 2022-12-01T10:04:01.6082443Z inflating: build/lib/libtorch_global_deps.so 2022-12-01T10:04:01.6089734Z inflating: build/lib/libcpuinfo.a 2022-12-01T10:04:01.6096729Z inflating: build/lib/libcpuinfo_internals.a 2022-12-01T10:04:01.6108978Z inflating: build/lib/libqnnpack.a 2022-12-01T10:04:01.6127642Z inflating: build/lib/libpytorch_qnnpack.a 2022-12-01T10:04:01.6129507Z inflating: build/lib/libnnpack_reference_layers.a 2022-12-01T10:04:01.6143983Z inflating: build/lib/libgmock.a 2022-12-01T10:04:01.6587848Z inflating: build/lib/libprotoc.a 2022-12-01T10:04:01.6588140Z inflating: build/lib/libgtest_main.a 2022-12-01T10:04:01.6588525Z inflating: build/lib/libbenchmark_main.a 2022-12-01T10:04:02.2846733Z inflating: build/lib/libdnnl.a 2022-12-01T10:04:02.2863566Z inflating: build/lib/libnnpack.a 2022-12-01T10:04:02.3368987Z inflating: build/lib/libtensorpipe.a 2022-12-01T10:04:02.4554010Z inflating: build/lib/libfbgemm.a 2022-12-01T10:04:02.5425769Z inflating: build/lib/libdnnl_graph.a 2022-12-01T10:04:02.5426188Z inflating: build/lib/libgmock_main.a 2022-12-01T10:04:02.5616362Z inflating: build/lib/libkineto.a 2022-12-01T10:04:02.5651315Z inflating: build/lib/libcaffe2_protos.a 2022-12-01T10:04:02.5688214Z inflating: build/lib/libonnx_proto.a 2022-12-01T10:04:02.5798830Z inflating: build/lib/libXNNPACK.a 2022-12-01T10:04:02.6323339Z inflating: build/lib/libonnx.a 2022-12-01T10:04:04.4280084Z inflating: build/lib/libtorch_cpu.so 2022-12-01T10:04:04.4280899Z inflating: build/lib/libtorch.so 2022-12-01T10:04:04.4299604Z inflating: build/lib/libjitbackend_test.so 2022-12-01T10:04:04.4346531Z inflating: build/lib/libtorchbind_test.so 2022-12-01T10:04:04.4370660Z inflating: build/lib/libbackend_with_compiler.so 2022-12-01T10:04:04.4374240Z inflating: build/lib/libshm.so 2022-12-01T10:04:04.5673251Z inflating: build/lib/libtorch_python.so 2022-12-01T10:04:04.5703972Z inflating: build/lib/libnnapi_backend.so 2022-12-01T10:04:04.5704210Z creating: build/bin/ 2022-12-01T10:04:04.5745361Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-12-01T10:04:04.5788494Z inflating: build/bin/c10_DeviceGuard_test 2022-12-01T10:04:04.5830517Z inflating: build/bin/c10_Device_test 2022-12-01T10:04:04.5879815Z inflating: build/bin/c10_DispatchKeySet_test 2022-12-01T10:04:04.5919780Z inflating: build/bin/c10_StreamGuard_test 2022-12-01T10:04:04.5961867Z inflating: build/bin/c10_SymInt_test 2022-12-01T10:04:04.6008141Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-12-01T10:04:04.6054825Z inflating: build/bin/c10_InlineStreamGuard_test 2022-12-01T10:04:04.6094928Z inflating: build/bin/c10_Array_test 2022-12-01T10:04:04.6142590Z inflating: build/bin/c10_SizesAndStrides_test 2022-12-01T10:04:04.6186775Z inflating: build/bin/c10_Bitset_test 2022-12-01T10:04:04.6227068Z inflating: build/bin/c10_ConstexprCrc_test 2022-12-01T10:04:04.6269730Z inflating: build/bin/c10_C++17_test 2022-12-01T10:04:04.6310660Z inflating: build/bin/c10_DeadlockDetection_test 2022-12-01T10:04:04.6352254Z inflating: build/bin/c10_Half_test 2022-12-01T10:04:04.6399565Z inflating: build/bin/c10_LeftRight_test 2022-12-01T10:04:04.6452020Z inflating: build/bin/c10_Metaprogramming_test 2022-12-01T10:04:04.6494355Z inflating: build/bin/c10_Synchronized_test 2022-12-01T10:04:04.6614513Z inflating: build/bin/c10_SmallVectorTest 2022-12-01T10:04:04.6662381Z inflating: build/bin/c10_ThreadLocal_test 2022-12-01T10:04:04.6704514Z inflating: build/bin/c10_TypeList_test 2022-12-01T10:04:04.6748384Z inflating: build/bin/c10_TypeIndex_test 2022-12-01T10:04:04.6788536Z inflating: build/bin/c10_TypeTraits_test 2022-12-01T10:04:04.6831536Z inflating: build/bin/c10_accumulate_test 2022-12-01T10:04:04.6878439Z inflating: build/bin/c10_bfloat16_test 2022-12-01T10:04:04.6923201Z inflating: build/bin/c10_complex_math_test 2022-12-01T10:04:04.6969800Z inflating: build/bin/c10_complex_test 2022-12-01T10:04:04.7013360Z inflating: build/bin/c10_exception_test 2022-12-01T10:04:04.7104803Z inflating: build/bin/c10_either_test 2022-12-01T10:04:04.7146673Z inflating: build/bin/c10_flags_test 2022-12-01T10:04:04.7287507Z inflating: build/bin/c10_intrusive_ptr_test 2022-12-01T10:04:04.7329855Z inflating: build/bin/c10_irange_test 2022-12-01T10:04:04.7378101Z inflating: build/bin/c10_logging_test 2022-12-01T10:04:04.7440721Z inflating: build/bin/c10_optional_test 2022-12-01T10:04:04.7486484Z inflating: build/bin/c10_registry_test 2022-12-01T10:04:04.7538593Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-12-01T10:04:04.7587968Z inflating: build/bin/c10_string_view_test 2022-12-01T10:04:04.7630702Z inflating: build/bin/c10_tempfile_test 2022-12-01T10:04:04.7678016Z inflating: build/bin/c10_typeid_test 2022-12-01T10:04:04.7725418Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-12-01T10:04:04.8134719Z inflating: build/bin/protoc-3.13.0.0 2022-12-01T10:04:04.8543622Z inflating: build/bin/protoc 2022-12-01T10:04:04.8789561Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-12-01T10:04:04.9064271Z inflating: build/bin/vec_test_all_types_AVX2 2022-12-01T10:04:04.9109217Z inflating: build/bin/FileStoreTest 2022-12-01T10:04:04.9159776Z inflating: build/bin/TCPStoreTest 2022-12-01T10:04:04.9204750Z inflating: build/bin/HashStoreTest 2022-12-01T10:04:04.9207287Z inflating: build/bin/example_allreduce 2022-12-01T10:04:04.9268274Z inflating: build/bin/Dict_test 2022-12-01T10:04:04.9322842Z inflating: build/bin/ProcessGroupGlooTest 2022-12-01T10:04:04.9366721Z inflating: build/bin/Dimname_test 2022-12-01T10:04:04.9419915Z inflating: build/bin/MaybeOwned_test 2022-12-01T10:04:04.9467684Z inflating: build/bin/NamedTensor_test 2022-12-01T10:04:04.9517001Z inflating: build/bin/apply_utils_test 2022-12-01T10:04:04.9566392Z inflating: build/bin/atest 2022-12-01T10:04:04.9611986Z inflating: build/bin/broadcast_test 2022-12-01T10:04:04.9663340Z inflating: build/bin/basic 2022-12-01T10:04:04.9712587Z inflating: build/bin/cpu_generator_test 2022-12-01T10:04:04.9756352Z inflating: build/bin/cpu_profiling_allocator_test 2022-12-01T10:04:04.9830740Z inflating: build/bin/cpu_rng_test 2022-12-01T10:04:04.9872767Z inflating: build/bin/dispatch_key_set_test 2022-12-01T10:04:04.9914643Z inflating: build/bin/dlconvertor_test 2022-12-01T10:04:04.9963684Z inflating: build/bin/extension_backend_test 2022-12-01T10:04:05.0010608Z inflating: build/bin/half_test 2022-12-01T10:04:05.0090177Z inflating: build/bin/ivalue_test 2022-12-01T10:04:05.0132115Z inflating: build/bin/lazy_tensor_test 2022-12-01T10:04:05.0177050Z inflating: build/bin/math_kernel_test 2022-12-01T10:04:05.0222286Z inflating: build/bin/memory_format_test 2022-12-01T10:04:05.0267044Z inflating: build/bin/memory_overlapping_test 2022-12-01T10:04:05.0310044Z inflating: build/bin/operator_name_test 2022-12-01T10:04:05.0354212Z inflating: build/bin/mobile_memory_cleanup 2022-12-01T10:04:05.0401788Z inflating: build/bin/native_test 2022-12-01T10:04:05.0446249Z inflating: build/bin/packedtensoraccessor_test 2022-12-01T10:04:05.0488538Z inflating: build/bin/operators_test 2022-12-01T10:04:05.0543713Z inflating: build/bin/pow_test 2022-12-01T10:04:05.0592207Z inflating: build/bin/quantized_test 2022-12-01T10:04:05.0634103Z inflating: build/bin/reduce_ops_test 2022-12-01T10:04:05.0676976Z inflating: build/bin/reportMemoryUsage_test 2022-12-01T10:04:05.0725086Z inflating: build/bin/scalar_tensor_test 2022-12-01T10:04:05.0774816Z inflating: build/bin/scalar_test 2022-12-01T10:04:05.0818876Z inflating: build/bin/stride_properties_test 2022-12-01T10:04:05.0885819Z inflating: build/bin/tensor_iterator_test 2022-12-01T10:04:05.0888324Z inflating: build/bin/thread_init_test 2022-12-01T10:04:05.0935531Z inflating: build/bin/type_ptr_test 2022-12-01T10:04:05.0982749Z inflating: build/bin/test_parallel 2022-12-01T10:04:05.1024508Z inflating: build/bin/variant_test 2022-12-01T10:04:05.1068484Z inflating: build/bin/undefined_tensor_test 2022-12-01T10:04:05.1119700Z inflating: build/bin/type_test 2022-12-01T10:04:05.1121195Z inflating: build/bin/verify_api_visibility 2022-12-01T10:04:05.1180029Z inflating: build/bin/vmap_test 2022-12-01T10:04:05.1222920Z inflating: build/bin/wrapdim_test 2022-12-01T10:04:05.1265946Z inflating: build/bin/weakref_test 2022-12-01T10:04:05.1316528Z inflating: build/bin/IListRef_test 2022-12-01T10:04:05.1408608Z inflating: build/bin/List_test 2022-12-01T10:04:05.1449973Z inflating: build/bin/xla_tensor_test 2022-12-01T10:04:05.1552727Z inflating: build/bin/kernel_function_legacy_test 2022-12-01T10:04:05.1634170Z inflating: build/bin/kernel_function_test 2022-12-01T10:04:05.1689363Z inflating: build/bin/KernelFunction_test 2022-12-01T10:04:05.1797470Z inflating: build/bin/kernel_lambda_legacy_test 2022-12-01T10:04:05.1884903Z inflating: build/bin/kernel_lambda_test 2022-12-01T10:04:05.1935651Z inflating: build/bin/kernel_stackbased_test 2022-12-01T10:04:05.1978162Z inflating: build/bin/CppSignature_test 2022-12-01T10:04:05.2059509Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-12-01T10:04:05.2100062Z inflating: build/bin/op_allowlist_test 2022-12-01T10:04:05.2144759Z inflating: build/bin/inline_container_test 2022-12-01T10:04:05.2388743Z inflating: build/bin/op_registration_test 2022-12-01T10:04:05.2436284Z inflating: build/bin/backend_fallback_test 2022-12-01T10:04:05.2450623Z inflating: build/bin/tutorial_tensorexpr 2022-12-01T10:04:05.2496377Z inflating: build/bin/test_dist_autograd 2022-12-01T10:04:05.2555610Z inflating: build/bin/test_cpp_rpc 2022-12-01T10:04:05.2558106Z inflating: build/bin/parallel_benchmark 2022-12-01T10:04:05.2616885Z inflating: build/bin/test_mobile_nnc 2022-12-01T10:04:05.2625998Z inflating: build/bin/aot_model_compiler_test 2022-12-01T10:04:05.2630950Z inflating: build/bin/torch_shm_manager 2022-12-01T10:04:05.2928527Z inflating: build/bin/test_lazy 2022-12-01T10:04:05.3611251Z inflating: build/bin/test_tensorexpr 2022-12-01T10:04:05.4623477Z inflating: build/bin/test_api 2022-12-01T10:04:05.5117439Z inflating: build/bin/test_jit 2022-12-01T10:04:05.5119031Z inflating: .pytorch-test-times.json 2022-12-01T10:04:05.5141624Z ##[group]Run df -H 2022-12-01T10:04:05.5141811Z df -H 2022-12-01T10:04:05.5152788Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:04:05.5152992Z env: 2022-12-01T10:04:05.5153168Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:04:05.5153352Z ##[endgroup] 2022-12-01T10:04:05.5282247Z Filesystem Size Used Avail Use% Mounted on 2022-12-01T10:04:05.5282583Z devtmpfs 8.2G 0 8.2G 0% /dev 2022-12-01T10:04:05.5282855Z tmpfs 8.2G 14M 8.2G 1% /dev/shm 2022-12-01T10:04:05.5283066Z tmpfs 8.2G 431k 8.2G 1% /run 2022-12-01T10:04:05.5283282Z tmpfs 8.2G 0 8.2G 0% /sys/fs/cgroup 2022-12-01T10:04:05.5283486Z /dev/nvme0n1p1 162G 22G 140G 14% / 2022-12-01T10:04:05.5302637Z ##[group]Run .github/scripts/parse_ref.py 2022-12-01T10:04:05.5302877Z .github/scripts/parse_ref.py 2022-12-01T10:04:05.5312788Z shell: /usr/bin/bash -e {0} 2022-12-01T10:04:05.5312974Z env: 2022-12-01T10:04:05.5313136Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:04:05.5313325Z ##[endgroup] 2022-12-01T10:04:05.5487975Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-12-01T10:04:05.5532135Z ##[group]Run set -x 2022-12-01T10:04:05.5532395Z set -x 2022-12-01T10:04:05.5532547Z  2022-12-01T10:04:05.5532741Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-12-01T10:04:05.5532999Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-12-01T10:04:05.5533243Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-12-01T10:04:05.5533479Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-12-01T10:04:05.5533747Z else 2022-12-01T10:04:05.5533936Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-12-01T10:04:05.5534134Z fi 2022-12-01T10:04:05.5534292Z  2022-12-01T10:04:05.5534512Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2022-12-01T10:04:05.5534748Z  2022-12-01T10:04:05.5534962Z # sanitize the input commit message and PR body here: 2022-12-01T10:04:05.5535174Z # 2022-12-01T10:04:05.5535450Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2022-12-01T10:04:05.5535807Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2022-12-01T10:04:05.5536116Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2022-12-01T10:04:05.5536345Z PR_BODY="${PR_BODY//[$'\n\r']}" 2022-12-01T10:04:05.5536532Z  2022-12-01T10:04:05.5536778Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2022-12-01T10:04:05.5537057Z # wreak havoc internally 2022-12-01T10:04:05.5537292Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2022-12-01T10:04:05.5537520Z export PR_BODY="${PR_BODY//[\'\"]}" 2022-12-01T10:04:05.5537707Z  2022-12-01T10:04:05.5537932Z # detached container should get cleaned up by teardown_ec2_linux 2022-12-01T10:04:05.5538215Z # TODO: Stop building test binaries as part of the build phase 2022-12-01T10:04:05.5538487Z # Used for GPU_FLAG since that doesn't play nice 2022-12-01T10:04:05.5538726Z # shellcheck disable=SC2086,SC2090 2022-12-01T10:04:05.5539030Z container_name=$(docker run \ 2022-12-01T10:04:05.5539222Z  ${GPU_FLAG:-} \ 2022-12-01T10:04:05.5539418Z  -e BUILD_ENVIRONMENT \ 2022-12-01T10:04:05.5539619Z  -e PR_NUMBER \ 2022-12-01T10:04:05.5539797Z  -e GITHUB_ACTIONS \ 2022-12-01T10:04:05.5539990Z  -e BASE_SHA \ 2022-12-01T10:04:05.5540170Z  -e BRANCH \ 2022-12-01T10:04:05.5540332Z  -e SHA1 \ 2022-12-01T10:04:05.5540518Z  -e AWS_DEFAULT_REGION \ 2022-12-01T10:04:05.5540717Z  -e IN_WHEEL_TEST \ 2022-12-01T10:04:05.5540894Z  -e SHARD_NUMBER \ 2022-12-01T10:04:05.5541082Z  -e TEST_CONFIG \ 2022-12-01T10:04:05.5541276Z  -e NUM_TEST_SHARDS \ 2022-12-01T10:04:05.5541519Z  -e PR_BODY \ 2022-12-01T10:04:05.5541698Z  -e COMMIT_MESSAGES \ 2022-12-01T10:04:05.5541909Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-12-01T10:04:05.5542136Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2022-12-01T10:04:05.5542331Z  -e PR_LABELS \ 2022-12-01T10:04:05.5542541Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-12-01T10:04:05.5542752Z  -e SCCACHE_BUCKET \ 2022-12-01T10:04:05.5542943Z  -e SCCACHE_S3_KEY_PREFIX \ 2022-12-01T10:04:05.5543143Z  -e XLA_CUDA \ 2022-12-01T10:04:05.5543351Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-12-01T10:04:05.5543587Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-12-01T10:04:05.5543821Z  --ulimit stack=10485760:83886080 \ 2022-12-01T10:04:05.5544050Z  --security-opt seccomp=unconfined \ 2022-12-01T10:04:05.5544275Z  --cap-add=SYS_PTRACE \ 2022-12-01T10:04:05.5544453Z  --ipc=host \ 2022-12-01T10:04:05.5544705Z  --shm-size="${SHM_SIZE}" \ 2022-12-01T10:04:05.5544894Z  --tty \ 2022-12-01T10:04:05.5545054Z  --detach \ 2022-12-01T10:04:05.5545251Z  --name="${container_name}" \ 2022-12-01T10:04:05.5545449Z  --user jenkins \ 2022-12-01T10:04:05.5545669Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-12-01T10:04:05.5545920Z  -w /var/lib/jenkins/workspace \ 2022-12-01T10:04:05.5546125Z  "${DOCKER_IMAGE}" 2022-12-01T10:04:05.5546315Z ) 2022-12-01T10:04:05.5546584Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2022-12-01T10:04:05.5556733Z shell: /usr/bin/bash -e {0} 2022-12-01T10:04:05.5556900Z env: 2022-12-01T10:04:05.5557073Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:04:05.5557311Z BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7 2022-12-01T10:04:05.5557529Z PR_NUMBER: 89997 2022-12-01T10:04:05.5557694Z BRANCH: pull/89997 2022-12-01T10:04:05.5557910Z SHA1: c13d400bffe90e16b96520bbc8a41a6f0c9cd584 2022-12-01T10:04:05.5558156Z BASE_SHA: ae2fe4033cf3b17259b17f351020b988fa893f91 2022-12-01T10:04:05.5558532Z PYTORCH_RETRY_TEST_CASES: 1 2022-12-01T10:04:05.5558742Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-12-01T10:04:05.5558945Z TEST_CONFIG: distributed 2022-12-01T10:04:05.5559141Z SHARD_NUMBER: 1 2022-12-01T10:04:05.5559316Z NUM_TEST_SHARDS: 2 2022-12-01T10:04:05.5559983Z PR_BODY: Link to landed master PR (if applicable): https://github.com/pytorch/pytorch/pull/88993 Criteria category: 1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.13 2: Prevents a crash I'm hoping this is a low-risk change, since it's just changing one method for its safer form. 2022-12-01T10:04:05.5560668Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-12-01T10:04:05.5560897Z SCCACHE_S3_KEY_PREFIX: pull 2022-12-01T10:04:05.5561083Z SHM_SIZE: 1g 2022-12-01T10:04:05.5561416Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b 2022-12-01T10:04:05.5561730Z XLA_CUDA: 2022-12-01T10:04:05.5561976Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-12-01T10:04:05.5562237Z ##[endgroup] 2022-12-01T10:04:05.5584785Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-12-01T10:04:05.5585377Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]] 2022-12-01T10:04:05.5585753Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-12-01T10:04:05.5588587Z ++ git cherry -v origin/master 2022-12-01T10:04:05.9278812Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290) 2022-12-01T10:04:05.9279273Z - f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385) 2022-12-01T10:04:05.9279720Z + 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484) 2022-12-01T10:04:05.9280234Z - c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377) 2022-12-01T10:04:05.9280958Z - 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689) 2022-12-01T10:04:05.9281614Z - 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725) 2022-12-01T10:04:05.9282404Z - f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844) 2022-12-01T10:04:05.9283048Z - 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs (#86851) 2022-12-01T10:04:05.9283975Z - 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784) 2022-12-01T10:04:05.9284561Z + d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333) 2022-12-01T10:04:05.9285555Z + 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693) 2022-12-01T10:04:05.9286293Z - 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895) 2022-12-01T10:04:05.9286806Z + f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904) 2022-12-01T10:04:05.9287267Z - 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903) 2022-12-01T10:04:05.9287903Z - 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726) 2022-12-01T10:04:05.9288361Z + 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902) 2022-12-01T10:04:05.9288883Z - aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921) 2022-12-01T10:04:05.9289461Z - de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933) 2022-12-01T10:04:05.9289872Z + 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803) 2022-12-01T10:04:05.9290243Z + 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934) 2022-12-01T10:04:05.9290753Z - f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516) 2022-12-01T10:04:05.9291343Z - c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201) 2022-12-01T10:04:05.9291777Z + eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140) 2022-12-01T10:04:05.9292184Z + d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132) 2022-12-01T10:04:05.9292642Z - 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925) 2022-12-01T10:04:05.9293140Z - 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924) 2022-12-01T10:04:05.9293671Z - 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509) 2022-12-01T10:04:05.9294039Z + d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672) 2022-12-01T10:04:05.9294571Z - 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923) 2022-12-01T10:04:05.9295096Z - ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122) 2022-12-01T10:04:05.9295625Z - e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144) 2022-12-01T10:04:05.9296035Z + 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261) 2022-12-01T10:04:05.9296563Z - 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337) 2022-12-01T10:04:05.9297071Z - 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307) 2022-12-01T10:04:05.9297553Z - 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376) 2022-12-01T10:04:05.9298069Z - 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support. (#86220) (#87379) 2022-12-01T10:04:05.9298629Z - 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380) 2022-12-01T10:04:05.9299201Z - aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381) 2022-12-01T10:04:05.9299739Z - 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386) 2022-12-01T10:04:05.9300229Z + ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079) 2022-12-01T10:04:05.9300799Z - d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384) 2022-12-01T10:04:05.9301307Z - 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in `bitwise_not` (#87286) (#87427) 2022-12-01T10:04:05.9301741Z - 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408) 2022-12-01T10:04:05.9302245Z - 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434) 2022-12-01T10:04:05.9302665Z + 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392) 2022-12-01T10:04:05.9303126Z - 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406) 2022-12-01T10:04:05.9303470Z + 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443) 2022-12-01T10:04:05.9303874Z + d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455) 2022-12-01T10:04:05.9313769Z - d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454) 2022-12-01T10:04:05.9314371Z - 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456) 2022-12-01T10:04:05.9314859Z - f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463) 2022-12-01T10:04:05.9315427Z - 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457) 2022-12-01T10:04:05.9315855Z + 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475) 2022-12-01T10:04:05.9338688Z - fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612) 2022-12-01T10:04:05.9339341Z - 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613) 2022-12-01T10:04:05.9339739Z - 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628) 2022-12-01T10:04:05.9340165Z - 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618) 2022-12-01T10:04:05.9340617Z - 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925) 2022-12-01T10:04:05.9340936Z + ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923) 2022-12-01T10:04:05.9341354Z - c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack' 2022-12-01T10:04:05.9386743Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support. (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack' 2022-12-01T10:04:05.9393623Z + PR_BODY='Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashI'\''m hoping this is a low-risk change, since it'\''s just changing one method for its safer form.' 2022-12-01T10:04:05.9424576Z + export 'COMMIT_MESSAGES=+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support. (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack' 2022-12-01T10:04:05.9442802Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support. (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack' 2022-12-01T10:04:05.9449627Z + export 'PR_BODY=Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.' 2022-12-01T10:04:05.9451452Z + PR_BODY='Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.' 2022-12-01T10:04:05.9452105Z +++ nproc --ignore=2 2022-12-01T10:04:05.9453630Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3591403534 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b 2022-12-01T10:04:16.9119152Z + container_name=ad94a0d9dab526da42383dc844e6795f91cab8542f2f3aa7f61579dc8525ac0c 2022-12-01T10:04:16.9122358Z ++ echo dist/torch-1.13.0a0+gitc13d400-cp37-cp37m-linux_x86_64.whl 2022-12-01T10:04:16.9123899Z + docker exec -t ad94a0d9dab526da42383dc844e6795f91cab8542f2f3aa7f61579dc8525ac0c sh -c 'pip install dist/torch-1.13.0a0+gitc13d400-cp37-cp37m-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh' 2022-12-01T10:04:17.3170133Z Processing ./dist/torch-1.13.0a0+gitc13d400-cp37-cp37m-linux_x86_64.whl 2022-12-01T10:04:17.3916323Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0a0+gitc13d400) (4.3.0) 2022-12-01T10:04:17.3929267Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0a0+gitc13d400) (3.3.0) 2022-12-01T10:04:17.3996464Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.7/site-packages (from opt-einsum>=3.3->torch==1.13.0a0+gitc13d400) (1.18.5) 2022-12-01T10:04:18.1792895Z Installing collected packages: torch 2022-12-01T10:04:24.4595951Z Successfully installed torch-1.13.0a0+gitc13d400 2022-12-01T10:04:24.5925864Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-12-01T10:04:24.6081232Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch 2022-12-01T10:04:24.6081896Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin 2022-12-01T10:04:24.6082429Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib 2022-12-01T10:04:24.6082794Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test 2022-12-01T10:04:24.6083015Z + BUILD_DIR=build 2022-12-01T10:04:24.6083211Z + BUILD_RENAMED_DIR=build_renamed 2022-12-01T10:04:24.6083411Z + BUILD_BIN_DIR=build/bin 2022-12-01T10:04:24.6083586Z + export VALGRIND=ON 2022-12-01T10:04:24.6083766Z + VALGRIND=ON 2022-12-01T10:04:24.6084021Z + [[ linux-focal-py3.7-gcc7 == *clang9* ]] 2022-12-01T10:04:24.6084318Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]] 2022-12-01T10:04:24.6084841Z ++ realpath build/custom_test_artifacts 2022-12-01T10:04:24.6090843Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2022-12-01T10:04:24.6093390Z ++ dirname .jenkins/pytorch/test.sh 2022-12-01T10:04:24.6098875Z + source .jenkins/pytorch/common.sh 2022-12-01T10:04:24.6102777Z +++ dirname .jenkins/pytorch/common.sh 2022-12-01T10:04:24.6107695Z ++ source .jenkins/pytorch/common_utils.sh 2022-12-01T10:04:24.6109236Z +++ declare -f -t trap_add 2022-12-01T10:04:24.6112881Z ++ set -ex 2022-12-01T10:04:24.6113669Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-12-01T10:04:24.6114010Z ++ BUILD_TEST_LIBTORCH=0 2022-12-01T10:04:24.6114238Z ++ [[ distributed == *xla* ]] 2022-12-01T10:04:24.6114627Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]] 2022-12-01T10:04:24.6114922Z ++ [[ linux-focal-py3.7-gcc7 == *linux-bionic* ]] 2022-12-01T10:04:24.6115228Z ++ [[ linux-focal-py3.7-gcc7 == *linux-focal* ]] 2022-12-01T10:04:24.6115440Z ++ which conda 2022-12-01T10:04:24.6121709Z /opt/conda/bin/conda 2022-12-01T10:04:24.6122539Z ++ conda install -q -y cmake 2022-12-01T10:04:27.3934288Z Collecting package metadata (current_repodata.json): ...working... done 2022-12-01T10:04:28.2463332Z Solving environment: ...working... done 2022-12-01T10:04:28.3338800Z 2022-12-01T10:04:28.3339397Z ## Package Plan ## 2022-12-01T10:04:28.3339626Z 2022-12-01T10:04:28.3339798Z environment location: /opt/conda 2022-12-01T10:04:28.3340153Z 2022-12-01T10:04:28.3340307Z added / updated specs: 2022-12-01T10:04:28.3340784Z - cmake 2022-12-01T10:04:28.3340935Z 2022-12-01T10:04:28.3340954Z 2022-12-01T10:04:28.3341067Z The following packages will be downloaded: 2022-12-01T10:04:28.3341211Z 2022-12-01T10:04:28.3341310Z package | build 2022-12-01T10:04:28.3341592Z ---------------------------|----------------- 2022-12-01T10:04:28.3341894Z bzip2-1.0.8 | h7b6447c_0 78 KB 2022-12-01T10:04:28.3342190Z c-ares-1.18.1 | h7f8727e_0 114 KB 2022-12-01T10:04:28.3342510Z ca-certificates-2022.10.11 | h06a4308_0 124 KB 2022-12-01T10:04:28.3342815Z certifi-2022.9.24 | py37h06a4308_0 154 KB 2022-12-01T10:04:28.3343112Z cmake-3.22.1 | h1fce559_0 7.3 MB 2022-12-01T10:04:28.3343400Z expat-2.4.9 | h6a678d5_0 156 KB 2022-12-01T10:04:28.3343683Z krb5-1.19.2 | hac12032_0 1.2 MB 2022-12-01T10:04:28.3344155Z libcurl-7.84.0 | h91b91d3_0 337 KB 2022-12-01T10:04:28.3344450Z libedit-3.1.20210910 | h7f8727e_0 166 KB 2022-12-01T10:04:28.3344734Z libev-4.33 | h7f8727e_1 111 KB 2022-12-01T10:04:28.3345004Z libgcc-ng-11.2.0 | h1234567_1 5.3 MB 2022-12-01T10:04:28.3345292Z libgomp-11.2.0 | h1234567_1 474 KB 2022-12-01T10:04:28.3345588Z libnghttp2-1.46.0 | hce63b2e_0 680 KB 2022-12-01T10:04:28.3345925Z libssh2-1.10.0 | h8f2d780_0 274 KB 2022-12-01T10:04:28.3346219Z libstdcxx-ng-11.2.0 | h1234567_1 4.7 MB 2022-12-01T10:04:28.3346507Z libuv-1.40.0 | h7b6447c_0 736 KB 2022-12-01T10:04:28.3346784Z lz4-c-1.9.3 | h295c915_1 185 KB 2022-12-01T10:04:28.3347051Z openssl-1.1.1s | h7f8727e_0 3.6 MB 2022-12-01T10:04:28.3347336Z rhash-1.4.1 | h3c74f83_1 203 KB 2022-12-01T10:04:28.3347615Z zstd-1.5.2 | ha4553b6_0 488 KB 2022-12-01T10:04:28.3347901Z ------------------------------------------------------------ 2022-12-01T10:04:28.3348142Z Total: 26.3 MB 2022-12-01T10:04:28.3348264Z 2022-12-01T10:04:28.3348377Z The following NEW packages will be INSTALLED: 2022-12-01T10:04:28.3348524Z 2022-12-01T10:04:28.3348790Z bzip2 pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0 None 2022-12-01T10:04:28.3349149Z c-ares pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0 None 2022-12-01T10:04:28.3349517Z cmake pkgs/main/linux-64::cmake-3.22.1-h1fce559_0 None 2022-12-01T10:04:28.3349880Z expat pkgs/main/linux-64::expat-2.4.9-h6a678d5_0 None 2022-12-01T10:04:28.3350222Z krb5 pkgs/main/linux-64::krb5-1.19.2-hac12032_0 None 2022-12-01T10:04:28.3350587Z libcurl pkgs/main/linux-64::libcurl-7.84.0-h91b91d3_0 None 2022-12-01T10:04:28.3350971Z libedit pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0 None 2022-12-01T10:04:28.3351336Z libev pkgs/main/linux-64::libev-4.33-h7f8727e_1 None 2022-12-01T10:04:28.3351700Z libnghttp2 pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0 None 2022-12-01T10:04:28.3352082Z libssh2 pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0 None 2022-12-01T10:04:28.3352443Z libuv pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0 None 2022-12-01T10:04:28.3352803Z lz4-c pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1 None 2022-12-01T10:04:28.3353144Z rhash pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1 None 2022-12-01T10:04:28.3353497Z zstd pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0 None 2022-12-01T10:04:28.3353644Z 2022-12-01T10:04:28.3353744Z The following packages will be UPDATED: 2022-12-01T10:04:28.3353880Z 2022-12-01T10:04:28.3354075Z ca-certificates 2022.07.19-h06a4308_0 --> 2022.10.11-h06a4308_0 None 2022-12-01T10:04:28.3354438Z certifi 2022.9.14-py37h06a4308_0 --> 2022.9.24-py37h06a4308_0 None 2022-12-01T10:04:28.3354782Z libgcc-ng 9.3.0-h5101ec6_17 --> 11.2.0-h1234567_1 None 2022-12-01T10:04:28.3355120Z libgomp 9.3.0-h5101ec6_17 --> 11.2.0-h1234567_1 None 2022-12-01T10:04:28.3355441Z libstdcxx-ng 9.3.0-hd4cf53a_17 --> 11.2.0-h1234567_1 None 2022-12-01T10:04:28.3355782Z openssl 1.1.1q-h7f8727e_0 --> 1.1.1s-h7f8727e_0 None 2022-12-01T10:04:28.3355930Z 2022-12-01T10:04:28.3355934Z 2022-12-01T10:04:29.4360746Z Preparing transaction: ...working... done 2022-12-01T10:04:29.8587889Z Verifying transaction: ...working... done 2022-12-01T10:04:30.6068260Z Executing transaction: ...working... done 2022-12-01T10:04:30.6902498Z Retrieving notices: ...working... done 2022-12-01T10:04:30.8584161Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]] 2022-12-01T10:04:30.8584536Z + echo 'Environment variables' 2022-12-01T10:04:30.8584743Z Environment variables 2022-12-01T10:04:30.8584903Z + env 2022-12-01T10:04:30.8590637Z INSTALLED_DB=yes 2022-12-01T10:04:30.8591262Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-12-01T10:04:30.8591690Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7 2022-12-01T10:04:30.8591916Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2022-12-01T10:04:30.8592132Z HOSTNAME=ad94a0d9dab5 2022-12-01T10:04:30.8592846Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_56104502-9092-4ab1-80db-a5a0b920460e 2022-12-01T10:04:30.8593192Z GITHUB_ACTION=__self 2022-12-01T10:04:30.8593469Z GITHUB_RUN_NUMBER=69319 2022-12-01T10:04:30.8593720Z TEST_CONFIG=distributed 2022-12-01T10:04:30.8593962Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2022-12-01T10:04:30.8594224Z GITHUB_TRIGGERING_ACTOR=charlie-wt 2022-12-01T10:04:30.8594424Z GITHUB_REF_TYPE=branch 2022-12-01T10:04:30.8594610Z TORCH_CUDA_ARCH_LIST=Maxwell 2022-12-01T10:04:30.8594833Z BASE_SHA=ae2fe4033cf3b17259b17f351020b988fa893f91 2022-12-01T10:04:30.8595034Z XLA_CUDA= 2022-12-01T10:04:30.8595638Z *** 2022-12-01T10:04:30.8595876Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2022-12-01T10:04:30.8596164Z GITHUB_ACTIONS=true 2022-12-01T10:04:30.8596435Z SHA1=c13d400bffe90e16b96520bbc8a41a6f0c9cd584 2022-12-01T10:04:30.8596763Z GITHUB_SHA=cdc064133ad5e5a46a756ee9218659e8f252e950 2022-12-01T10:04:30.8597136Z UCC_HOME=/usr 2022-12-01T10:04:30.8597487Z GITHUB_REF=refs/pull/89997/merge 2022-12-01T10:04:30.8597816Z SHARD_NUMBER=1 2022-12-01T10:04:30.8598143Z GITHUB_REF_PROTECTED=false 2022-12-01T10:04:30.8598670Z HOME=/var/lib/jenkins 2022-12-01T10:04:30.8599028Z GITHUB_API_URL=https://api.github.com 2022-12-01T10:04:30.8599252Z INSTALLED_THRIFT= 2022-12-01T10:04:30.8599497Z UCX_COMMIT= 2022-12-01T10:04:30.8599793Z SCCACHE_S3_KEY_PREFIX=pull 2022-12-01T10:04:30.8600124Z NUM_TEST_SHARDS=2 2022-12-01T10:04:30.8600443Z UCX_HOME=/usr 2022-12-01T10:04:30.8600984Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_56104502-9092-4ab1-80db-a5a0b920460e 2022-12-01T10:04:30.8601278Z PYTORCH_RETRY_TEST_CASES=1 2022-12-01T10:04:30.8601861Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_56104502-9092-4ab1-80db-a5a0b920460e 2022-12-01T10:04:30.8602565Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2022-12-01T10:04:30.8602969Z GITHUB_EVENT_NAME=pull_request 2022-12-01T10:04:30.8603290Z GITHUB_RUN_ID=3591403534 2022-12-01T10:04:30.8603992Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_56104502-9092-4ab1-80db-a5a0b920460e 2022-12-01T10:04:30.8604528Z GITHUB_ACTOR=charlie-wt 2022-12-01T10:04:30.8604818Z PR_NUMBER=89997 2022-12-01T10:04:30.8605132Z DESIRED_CUDA= 2022-12-01T10:04:30.8605450Z GITHUB_RUN_ATTEMPT=1 2022-12-01T10:04:30.8605739Z VALGRIND=ON 2022-12-01T10:04:30.8606137Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2022-12-01T10:04:30.8606510Z TERM=xterm 2022-12-01T10:04:30.8606806Z INSTALLED_VISION=yes 2022-12-01T10:04:30.8607136Z BRANCH=pull/89997 2022-12-01T10:04:30.8607460Z OPENSSL_ROOT_DIR=/opt/openssl 2022-12-01T10:04:30.8607787Z CUDA_PATH=/usr/local/cuda 2022-12-01T10:04:30.8608433Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2022-12-01T10:04:30.8608952Z GITHUB_SERVER_URL=https://github.com 2022-12-01T10:04:30.8609316Z UCC_COMMIT= 2022-12-01T10:04:30.8609618Z INSTALLED_ANDROID= 2022-12-01T10:04:30.8609905Z SHLVL=1 2022-12-01T10:04:30.8610186Z MAX_JOBS=6 2022-12-01T10:04:30.8631948Z COMMIT_MESSAGES=+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support. (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack 2022-12-01T10:04:30.8642660Z GITHUB_REF_NAME=89997/merge 2022-12-01T10:04:30.8643308Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2022-12-01T10:04:30.8643762Z GITHUB_JOB=test 2022-12-01T10:04:30.8644100Z GITHUB_REPOSITORY=pytorch/pytorch 2022-12-01T10:04:30.8644446Z GITHUB_RETENTION_DAYS=90 2022-12-01T10:04:30.8644787Z OPENSSL_DIR=/opt/openssl 2022-12-01T10:04:30.8645133Z GITHUB_ACTION_REPOSITORY= 2022-12-01T10:04:30.8645700Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-12-01T10:04:30.8646199Z GITHUB_BASE_REF=release/1.13 2022-12-01T10:04:30.8646513Z CI=true 2022-12-01T10:04:30.8646831Z GITHUB_REPOSITORY_OWNER=pytorch 2022-12-01T10:04:30.8647172Z INSTALLED_PROTOBUF=yes 2022-12-01T10:04:30.8647636Z GITHUB_HEAD_REF=release/1.13-callstack 2022-12-01T10:04:30.8647982Z GITHUB_ACTION_REF= 2022-12-01T10:04:30.8648438Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2022-12-01T10:04:30.8648848Z GITHUB_WORKFLOW=pull 2022-12-01T10:04:30.8649193Z DEBIAN_FRONTEND=noninteractive 2022-12-01T10:04:30.8649914Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_56104502-9092-4ab1-80db-a5a0b920460e 2022-12-01T10:04:30.8651780Z PR_BODY=Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form. 2022-12-01T10:04:30.8652813Z _=/usr/bin/env 2022-12-01T10:04:30.8653207Z + echo 'Testing pytorch' 2022-12-01T10:04:30.8653534Z Testing pytorch 2022-12-01T10:04:30.8653860Z + export LANG=C.UTF-8 2022-12-01T10:04:30.8654193Z + LANG=C.UTF-8 2022-12-01T10:04:30.8654487Z + PR_NUMBER=89997 2022-12-01T10:04:30.8654788Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-12-01T10:04:30.8655174Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2022-12-01T10:04:30.8655636Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-12-01T10:04:30.8656123Z + [[ distributed == \s\l\o\w ]] 2022-12-01T10:04:30.8656606Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]] 2022-12-01T10:04:30.8657096Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]] 2022-12-01T10:04:30.8657554Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-12-01T10:04:30.8658021Z + [[ linux-focal-py3.7-gcc7 == *cuda11* ]] 2022-12-01T10:04:30.8658409Z + [[ distributed == *crossref* ]] 2022-12-01T10:04:30.8658752Z + [[ distributed == *dynamo* ]] 2022-12-01T10:04:30.8659222Z + [[ -n 89997 ]] 2022-12-01T10:04:30.8659664Z + [[ -z '' ]] 2022-12-01T10:04:30.8660023Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1 2022-12-01T10:04:30.8660417Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1 2022-12-01T10:04:30.8660894Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-12-01T10:04:30.8661378Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]] 2022-12-01T10:04:30.8661810Z + pip_install --user ninja 2022-12-01T10:04:30.8662290Z + pip install --progress-bar off --user ninja 2022-12-01T10:04:31.2642333Z Collecting ninja 2022-12-01T10:04:31.2824230Z Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB) 2022-12-01T10:04:32.0049606Z Installing collected packages: ninja 2022-12-01T10:04:32.0144517Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-12-01T10:04:32.0145244Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-12-01T10:04:32.0199305Z Successfully installed ninja-1.11.1 2022-12-01T10:04:32.0819053Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-12-01T10:04:32.0819665Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-12-01T10:04:32.0820325Z + [[ linux-focal-py3.7-gcc7 == *asan* ]] 2022-12-01T10:04:32.0820640Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-12-01T10:04:32.0820883Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2022-12-01T10:04:32.0824802Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]] 2022-12-01T10:04:32.0836641Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]] 2022-12-01T10:04:32.0836994Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]] 2022-12-01T10:04:32.0839923Z + cd test 2022-12-01T10:04:32.0840363Z + python -c 'import torch; print(torch.__config__.show())' 2022-12-01T10:04:32.8441155Z PyTorch built with: 2022-12-01T10:04:32.8441702Z - GCC 7.5 2022-12-01T10:04:32.8442066Z - C++ Version: 201402 2022-12-01T10:04:32.8442833Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-12-01T10:04:32.8443535Z - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) 2022-12-01T10:04:32.8443834Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-12-01T10:04:32.8444113Z - LAPACK is enabled (usually provided by MKL) 2022-12-01T10:04:32.8444369Z - NNPACK is enabled 2022-12-01T10:04:32.8444587Z - CPU capability usage: AVX2 2022-12-01T10:04:32.8447118Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2022-12-01T10:04:32.8448865Z 2022-12-01T10:04:32.9754866Z + cd test 2022-12-01T10:04:32.9755492Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-12-01T10:04:33.7201979Z ATen/Parallel: 2022-12-01T10:04:33.7202244Z at::get_num_threads() : 4 2022-12-01T10:04:33.7202446Z at::get_num_interop_threads() : 4 2022-12-01T10:04:33.7202679Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-12-01T10:04:33.7202879Z omp_get_max_threads() : 4 2022-12-01T10:04:33.7203380Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-12-01T10:04:33.7203670Z mkl_get_max_threads() : 4 2022-12-01T10:04:33.7204016Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) 2022-12-01T10:04:33.7204288Z std::thread::hardware_concurrency() : 8 2022-12-01T10:04:33.7204485Z Environment variables: 2022-12-01T10:04:33.7204683Z OMP_NUM_THREADS : [not set] 2022-12-01T10:04:33.7204881Z MKL_NUM_THREADS : [not set] 2022-12-01T10:04:33.7205070Z ATen parallel backend: OpenMP 2022-12-01T10:04:33.7205195Z 2022-12-01T10:04:33.8535637Z + [[ distributed == *backward* ]] 2022-12-01T10:04:33.8536029Z + [[ distributed == *xla* ]] 2022-12-01T10:04:33.8536297Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-12-01T10:04:33.8536736Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]] 2022-12-01T10:04:33.8536971Z + [[ distributed == distributed ]] 2022-12-01T10:04:33.8537158Z + install_torchdynamo 2022-12-01T10:04:33.8537397Z + local commit 2022-12-01T10:04:33.8539890Z ++ get_pinned_commit torchdynamo 2022-12-01T10:04:33.8540347Z ++ cat .github/ci_commit_pins/torchdynamo.txt 2022-12-01T10:04:33.8550463Z + commit=6ead5cae0d1234aa64db06fe230ef56e12ec76fe 2022-12-01T10:04:33.8551055Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe 2022-12-01T10:04:33.8551617Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe 2022-12-01T10:04:34.1960098Z Collecting git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe 2022-12-01T10:04:34.1965383Z Cloning https://github.com/pytorch/torchdynamo.git (to revision 6ead5cae0d1234aa64db06fe230ef56e12ec76fe) to /tmp/pip-req-build-zx0b58eb 2022-12-01T10:04:34.1994055Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-zx0b58eb 2022-12-01T10:04:34.7309481Z Running command git rev-parse -q --verify 'sha^6ead5cae0d1234aa64db06fe230ef56e12ec76fe' 2022-12-01T10:04:34.7341595Z Running command git fetch -q https://github.com/pytorch/torchdynamo.git 6ead5cae0d1234aa64db06fe230ef56e12ec76fe 2022-12-01T10:04:34.9715066Z Running command git checkout -q 6ead5cae0d1234aa64db06fe230ef56e12ec76fe 2022-12-01T10:04:35.4287315Z Resolved https://github.com/pytorch/torchdynamo.git to commit 6ead5cae0d1234aa64db06fe230ef56e12ec76fe 2022-12-01T10:04:36.8109337Z Preparing metadata (setup.py) ... [?25l- done 2022-12-01T10:04:36.8178229Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+gitc13d400) 2022-12-01T10:04:36.8182292Z Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.18.5) 2022-12-01T10:04:36.8533405Z Collecting tabulate 2022-12-01T10:04:36.8672522Z Downloading tabulate-0.9.0-py3-none-any.whl (35 kB) 2022-12-01T10:04:36.8735246Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (6.0) 2022-12-01T10:04:36.8738915Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.10.1) 2022-12-01T10:04:36.8769965Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0) 2022-12-01T10:04:36.8795234Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1) 2022-12-01T10:04:36.8886140Z Building wheels for collected packages: torchdynamo 2022-12-01T10:04:40.2859893Z Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - done 2022-12-01T10:04:40.2939161Z [?25h Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp37-cp37m-linux_x86_64.whl size=2686246 sha256=1f53b84c08e9a89b53e9036ad755a87da413ed5ef7fa7714a7c58f29fc941912 2022-12-01T10:04:40.2941193Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/be/04/a5/adc64d091c88f64465d8ee6f21d5717177ec59bba7342b9732 2022-12-01T10:04:40.2964069Z Successfully built torchdynamo 2022-12-01T10:04:41.0004336Z Installing collected packages: tabulate, torchdynamo 2022-12-01T10:04:41.2719136Z Successfully installed tabulate-0.9.0 torchdynamo-1.13.0.dev0 2022-12-01T10:04:41.3500333Z + test_distributed 2022-12-01T10:04:41.3500776Z + echo 'Testing distributed python tests' 2022-12-01T10:04:41.3501012Z Testing distributed python tests 2022-12-01T10:04:41.3501346Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose 2022-12-01T10:04:42.4698991Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' 2022-12-01T10:04:42.5317733Z Ignoring disabled issues: [] 2022-12-01T10:04:42.5454305Z Found test time stats from artifacts 2022-12-01T10:04:42.5462028Z Selected tests: 2022-12-01T10:04:42.5462364Z distributed/rpc/test_tensorpipe_agent 2022-12-01T10:04:42.5462713Z distributed/rpc/test_faulty_agent 2022-12-01T10:04:42.5463036Z distributed/fsdp/test_fsdp_core 2022-12-01T10:04:42.5463350Z distributed/fsdp/test_fsdp_summon_full_params 2022-12-01T10:04:42.5463672Z distributed/test_c10d_pypg 2022-12-01T10:04:42.5464027Z distributed/test_c10d_nccl 2022-12-01T10:04:42.5464409Z distributed/fsdp/test_fsdp_checkpoint 2022-12-01T10:04:42.5464692Z distributed/fsdp/test_fsdp_misc 2022-12-01T10:04:42.5464894Z distributed/test_store 2022-12-01T10:04:42.5465085Z distributed/test_c10d_spawn_gloo 2022-12-01T10:04:42.5465501Z distributed/fsdp/test_fsdp_freezing_weights 2022-12-01T10:04:42.5465942Z distributed/fsdp/test_fsdp_exec_order 2022-12-01T10:04:42.5466298Z distributed/algorithms/test_join 2022-12-01T10:04:42.5466619Z distributed/fsdp/test_fsdp_meta 2022-12-01T10:04:42.5467000Z distributed/fsdp/test_fsdp_ignored_modules 2022-12-01T10:04:42.5467421Z distributed/test_c10d_object_collectives 2022-12-01T10:04:42.5467740Z distributed/fsdp/test_fsdp_apply 2022-12-01T10:04:42.5468048Z distributed/elastic/timer/local_timer_test 2022-12-01T10:04:42.5468325Z distributed/fsdp/test_fsdp_memory 2022-12-01T10:04:42.5468697Z distributed/fsdp/test_fsdp_input 2022-12-01T10:04:42.5469110Z distributed/fsdp/test_fsdp_pure_fp16 2022-12-01T10:04:42.5469418Z distributed/fsdp/test_fsdp_multiple_wrapping 2022-12-01T10:04:42.5469660Z distributed/_shard/sharding_spec/test_sharding_spec 2022-12-01T10:04:42.5469909Z distributed/fsdp/test_fsdp_traversal 2022-12-01T10:04:42.5470132Z distributed/elastic/utils/util_test 2022-12-01T10:04:42.5470333Z distributed/fsdp/test_fsdp_fx 2022-12-01T10:04:42.5470547Z distributed/elastic/metrics/api_test 2022-12-01T10:04:42.5470759Z distributed/test_launcher 2022-12-01T10:04:42.5470984Z distributed/_shard/sharding_plan/test_sharding_plan 2022-12-01T10:04:42.5471252Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-12-01T10:04:42.5473741Z distributed/_shard/test_partial_tensor 2022-12-01T10:04:42.5474281Z distributed/_shard/sharded_tensor/ops/test_init 2022-12-01T10:04:42.5474842Z distributed/_shard/sharded_optim/test_sharded_optim 2022-12-01T10:04:42.5475419Z distributed/_shard/sharded_tensor/ops/test_embedding 2022-12-01T10:04:42.5476242Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-12-01T10:04:42.5476862Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2022-12-01T10:04:42.5477416Z distributed/_shard/sharded_tensor/ops/test_softmax 2022-12-01T10:04:42.5477973Z distributed/_shard/checkpoint/test_checkpoint 2022-12-01T10:04:42.5478713Z distributed/_shard/checkpoint/test_file_system_checkpoint_cpu 2022-12-01T10:04:42.5479266Z distributed/_shard/checkpoint/test_utils 2022-12-01T10:04:42.5479796Z distributed/_shard/test_replicated_tensor 2022-12-01T10:04:42.5480415Z distributed/elastic/events/lib_test 2022-12-01T10:04:42.5480852Z distributed/fsdp/test_flatten_params_wrapper 2022-12-01T10:04:42.5481296Z distributed/fsdp/test_shard_utils 2022-12-01T10:04:42.5481600Z distributed/pipeline/sync/skip/test_gpipe 2022-12-01T10:04:42.5481827Z distributed/pipeline/sync/skip/test_leak 2022-12-01T10:04:42.5482072Z distributed/pipeline/sync/skip/test_stash_pop 2022-12-01T10:04:42.5482345Z distributed/pipeline/sync/skip/test_verify_skippables 2022-12-01T10:04:42.5482598Z distributed/pipeline/sync/test_bugs 2022-12-01T10:04:42.5482810Z distributed/pipeline/sync/test_copy 2022-12-01T10:04:42.5483048Z distributed/pipeline/sync/test_dependency 2022-12-01T10:04:42.5483290Z distributed/pipeline/sync/test_microbatch 2022-12-01T10:04:42.5483507Z distributed/pipeline/sync/test_pipe 2022-12-01T10:04:42.5483738Z distributed/pipeline/sync/test_stream 2022-12-01T10:04:42.5483971Z distributed/pipeline/sync/test_worker 2022-12-01T10:04:42.7057748Z Prioritized test from test file changes. 2022-12-01T10:04:42.7058369Z reordering tests for PR: 2022-12-01T10:04:42.7060748Z prioritized: ['distributed/fsdp/test_fsdp_core', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/test_c10d_nccl', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_fsdp_misc', 'distributed/test_store', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/fsdp/test_fsdp_apply', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_input', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_fx', 'distributed/_shard/checkpoint/test_checkpoint', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu', 'distributed/_shard/checkpoint/test_utils', 'distributed/fsdp/test_flatten_params_wrapper'] 2022-12-01T10:04:42.7064087Z the rest: ['distributed/rpc/test_tensorpipe_agent', 'distributed/rpc/test_faulty_agent', 'distributed/test_c10d_pypg', 'distributed/test_c10d_spawn_gloo', 'distributed/algorithms/test_join', 'distributed/test_c10d_object_collectives', 'distributed/elastic/timer/local_timer_test', 'distributed/elastic/utils/util_test', 'distributed/elastic/metrics/api_test', 'distributed/test_launcher', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_shard/test_partial_tensor', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/_shard/test_replicated_tensor', 'distributed/elastic/events/lib_test', 'distributed/fsdp/test_shard_utils', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_worker'] 2022-12-01T10:04:42.7065828Z 2022-12-01T10:04:42.7066230Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2022-12-01T10:04:42.7243610Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2022-12-01T10:04:42.7390119Z parallel (file granularity) tests: 2022-12-01T10:04:42.7390664Z 2022-12-01T10:04:42.7390985Z serial (file granularity) tests: 2022-12-01T10:04:42.7391347Z distributed/fsdp/test_fsdp_core 2022-12-01T10:04:42.7391703Z distributed/fsdp/test_fsdp_summon_full_params 2022-12-01T10:04:42.7392025Z distributed/test_c10d_nccl 2022-12-01T10:04:42.7392356Z distributed/fsdp/test_fsdp_checkpoint 2022-12-01T10:04:42.7392759Z distributed/fsdp/test_fsdp_misc 2022-12-01T10:04:42.7393134Z distributed/test_store 2022-12-01T10:04:42.7393473Z distributed/fsdp/test_fsdp_freezing_weights 2022-12-01T10:04:42.7393852Z distributed/fsdp/test_fsdp_exec_order 2022-12-01T10:04:42.7394219Z distributed/fsdp/test_fsdp_meta 2022-12-01T10:04:42.7394578Z distributed/fsdp/test_fsdp_ignored_modules 2022-12-01T10:04:42.7394952Z distributed/fsdp/test_fsdp_apply 2022-12-01T10:04:42.7395314Z distributed/fsdp/test_fsdp_memory 2022-12-01T10:04:42.7395660Z distributed/fsdp/test_fsdp_input 2022-12-01T10:04:42.7396059Z distributed/fsdp/test_fsdp_pure_fp16 2022-12-01T10:04:42.7396466Z distributed/fsdp/test_fsdp_multiple_wrapping 2022-12-01T10:04:42.7396941Z distributed/_shard/sharding_spec/test_sharding_spec 2022-12-01T10:04:42.7397345Z distributed/fsdp/test_fsdp_traversal 2022-12-01T10:04:42.7397563Z distributed/fsdp/test_fsdp_fx 2022-12-01T10:04:42.7397793Z distributed/_shard/checkpoint/test_checkpoint 2022-12-01T10:04:42.7398050Z distributed/_shard/checkpoint/test_file_system_checkpoint_cpu 2022-12-01T10:04:42.7398534Z distributed/_shard/checkpoint/test_utils 2022-12-01T10:04:42.7398783Z distributed/fsdp/test_flatten_params_wrapper 2022-12-01T10:04:42.7399010Z distributed/rpc/test_tensorpipe_agent 2022-12-01T10:04:42.7399235Z distributed/rpc/test_faulty_agent 2022-12-01T10:04:42.7399445Z distributed/test_c10d_pypg 2022-12-01T10:04:42.7399642Z distributed/test_c10d_spawn_gloo 2022-12-01T10:04:42.7399855Z distributed/algorithms/test_join 2022-12-01T10:04:42.7400078Z distributed/test_c10d_object_collectives 2022-12-01T10:04:42.7400298Z distributed/elastic/timer/local_timer_test 2022-12-01T10:04:42.7400530Z distributed/elastic/utils/util_test 2022-12-01T10:04:42.7400753Z distributed/elastic/metrics/api_test 2022-12-01T10:04:42.7400965Z distributed/test_launcher 2022-12-01T10:04:42.7401185Z distributed/_shard/sharding_plan/test_sharding_plan 2022-12-01T10:04:42.7401452Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-12-01T10:04:42.7401696Z distributed/_shard/test_partial_tensor 2022-12-01T10:04:42.7401923Z distributed/_shard/sharded_tensor/ops/test_init 2022-12-01T10:04:42.7402181Z distributed/_shard/sharded_optim/test_sharded_optim 2022-12-01T10:04:42.7402446Z distributed/_shard/sharded_tensor/ops/test_embedding 2022-12-01T10:04:42.7402707Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-12-01T10:04:42.7402984Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2022-12-01T10:04:42.7403250Z distributed/_shard/sharded_tensor/ops/test_softmax 2022-12-01T10:04:42.7403483Z distributed/_shard/test_replicated_tensor 2022-12-01T10:04:42.7403717Z distributed/elastic/events/lib_test 2022-12-01T10:04:42.7403937Z distributed/fsdp/test_shard_utils 2022-12-01T10:04:42.7404156Z distributed/pipeline/sync/skip/test_gpipe 2022-12-01T10:04:42.7404392Z distributed/pipeline/sync/skip/test_leak 2022-12-01T10:04:42.7404635Z distributed/pipeline/sync/skip/test_stash_pop 2022-12-01T10:04:42.7404896Z distributed/pipeline/sync/skip/test_verify_skippables 2022-12-01T10:04:42.7405127Z distributed/pipeline/sync/test_bugs 2022-12-01T10:04:42.7405350Z distributed/pipeline/sync/test_copy 2022-12-01T10:04:42.7405705Z distributed/pipeline/sync/test_dependency 2022-12-01T10:04:42.7405930Z distributed/pipeline/sync/test_microbatch 2022-12-01T10:04:42.7406162Z distributed/pipeline/sync/test_pipe 2022-12-01T10:04:42.7406389Z distributed/pipeline/sync/test_stream 2022-12-01T10:04:42.7406606Z distributed/pipeline/sync/test_worker 2022-12-01T10:04:43.9028840Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' 2022-12-01T10:04:43.9129306Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' 2022-12-01T10:04:43.9636075Z Ignoring disabled issues: [] 2022-12-01T10:04:43.9729692Z Ignoring disabled issues: [] 2022-12-01T10:04:44.2343399Z Running distributed/fsdp/test_fsdp_core ... [2022-12-01 10:04:44.233991] 2022-12-01T10:04:44.2346386Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:04:44.234416] 2022-12-01T10:05:58.7662321Z 2022-12-01T10:05:58.7662889Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_core 2022-12-01T10:05:58.7696459Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_core_03u0b8q1) 2022-12-01T10:05:58.7697802Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core 2022-12-01T10:05:58.7698234Z 2022-12-01T10:05:58.7698475Z Running tests... 2022-12-01T10:05:58.7699177Z ---------------------------------------------------------------------- 2022-12-01T10:05:58.7701819Z test_pre_backward_hook_registration_after_state_dict (__main__.TestHooks) 2022-12-01T10:05:58.7702987Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 734 2022-12-01T10:05:58.7703629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 735 2022-12-01T10:05:58.7704342Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 736 2022-12-01T10:05:58.7712097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 737 2022-12-01T10:05:58.7712924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7713503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7714632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7715677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7716198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7716955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7717463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7718142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7718817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7719347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7720233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7720932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7721271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7721989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7722533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7723263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7724088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7724710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7725153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7725579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7725915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7726443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7727087Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7729658Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7730362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7730877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7731344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7731809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7732214Z dist init r=2, world=4 2022-12-01T10:05:58.7732517Z dist init r=0, world=4 2022-12-01T10:05:58.7732794Z dist init r=1, world=4 2022-12-01T10:05:58.7733076Z dist init r=3, world=4 2022-12-01T10:05:58.7733418Z skip: Need at least 2 CUDA devices (1.538s) 2022-12-01T10:05:58.7733925Z test_pre_backward_hook_registration_cuda_first_False (__main__.TestHooks) 2022-12-01T10:05:58.7734875Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 790 2022-12-01T10:05:58.7735351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 791 2022-12-01T10:05:58.7735675Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 792 2022-12-01T10:05:58.7736000Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 793 2022-12-01T10:05:58.7736448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7736783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7737218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7737554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7737894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7738430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7738763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7739184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7739533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7739869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7740362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7740683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7741196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7741547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7741869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7742367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7742869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7743239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7743654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7744001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7744337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7744839Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7745345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7745870Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7746262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7746613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7746944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7747282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7747574Z dist init r=2, world=4 2022-12-01T10:05:58.7747853Z dist init r=0, world=4 2022-12-01T10:05:58.7748161Z dist init r=3, world=4 2022-12-01T10:05:58.7748452Z dist init r=1, world=4 2022-12-01T10:05:58.7748758Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.7749184Z test_pre_backward_hook_registration_cuda_first_True (__main__.TestHooks) 2022-12-01T10:05:58.7749764Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 846 2022-12-01T10:05:58.7750170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 847 2022-12-01T10:05:58.7750481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 848 2022-12-01T10:05:58.7750801Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 849 2022-12-01T10:05:58.7756107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7760803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7761515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7781386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7782397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7782914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7783260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7783698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7784032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7784372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7784965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7785298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7785716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7786062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7786456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7786916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7787244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7787675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7788022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7788342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7788844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7789371Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7789898Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7790394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7790779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7791136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7791469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7791817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7792071Z dist init r=1, world=4 2022-12-01T10:05:58.7792253Z dist init r=0, world=4 2022-12-01T10:05:58.7792417Z dist init r=3, world=4 2022-12-01T10:05:58.7792595Z dist init r=2, world=4 2022-12-01T10:05:58.7792804Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.7793084Z test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks) 2022-12-01T10:05:58.7793494Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 902 2022-12-01T10:05:58.7793879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 903 2022-12-01T10:05:58.7794195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 904 2022-12-01T10:05:58.7794515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 905 2022-12-01T10:05:58.7794963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7795295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7795712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7796063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7796399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7796866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7797181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7797643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7797989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7798475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7798949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7799335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7799758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7800088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7800423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7800965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7801294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7801710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7802056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7802390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7802877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7803399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7803918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7804436Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7804806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7805153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7805496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7805844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7806091Z dist init r=0, world=4 2022-12-01T10:05:58.7806275Z dist init r=3, world=4 2022-12-01T10:05:58.7806452Z dist init r=1, world=4 2022-12-01T10:05:58.7806616Z dist init r=2, world=4 2022-12-01T10:05:58.7806822Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.7807113Z test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks) 2022-12-01T10:05:58.7807501Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 958 2022-12-01T10:05:58.7807895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 959 2022-12-01T10:05:58.7808218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 960 2022-12-01T10:05:58.7808540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 961 2022-12-01T10:05:58.7808981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7809313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7809745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7810078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7810464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7810940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7811270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7811682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7812060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7812394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7812845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7813175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7813602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7813945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7814260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7814730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7815059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7815481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7815806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7816138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7816633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7817136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7817648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7818162Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7818614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7818950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7819291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7819631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7819897Z dist init r=3, world=4 2022-12-01T10:05:58.7820068Z dist init r=1, world=4 2022-12-01T10:05:58.7820247Z dist init r=0, world=4 2022-12-01T10:05:58.7820425Z dist init r=2, world=4 2022-12-01T10:05:58.7820617Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:05:58.7820908Z test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks) 2022-12-01T10:05:58.7821317Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1014 2022-12-01T10:05:58.7821696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1015 2022-12-01T10:05:58.7822020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1016 2022-12-01T10:05:58.7822340Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1017 2022-12-01T10:05:58.7822795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7823154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7823583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7823932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7824255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7824757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7825085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7825512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7825843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7826177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7826644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7826971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7827381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7827726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7828058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7828537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7829033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7829364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7829790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7830122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7830456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7830950Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7831472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7831974Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7832360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7832709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7833041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7833385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7833643Z dist init r=3, world=4 2022-12-01T10:05:58.7833823Z dist init r=0, world=4 2022-12-01T10:05:58.7833989Z dist init r=2, world=4 2022-12-01T10:05:58.7834174Z dist init r=1, world=4 2022-12-01T10:05:58.7834379Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.7834656Z test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks) 2022-12-01T10:05:58.7835057Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1070 2022-12-01T10:05:58.7835446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1071 2022-12-01T10:05:58.7835803Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1072 2022-12-01T10:05:58.7836129Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1073 2022-12-01T10:05:58.7836581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7836916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7837331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7837717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7838059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7838764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7839084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7839509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7839857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7840179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7840643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7840973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7841397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7841728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7842059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7842556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7843051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7843367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7843797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7844179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7844500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7844998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7845516Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7846029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7846430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7846778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7847122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7847463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7847708Z dist init r=0, world=4 2022-12-01T10:05:58.7847889Z dist init r=2, world=4 2022-12-01T10:05:58.7848067Z dist init r=1, world=4 2022-12-01T10:05:58.7848230Z dist init r=3, world=4 2022-12-01T10:05:58.7848436Z skip: Need at least 2 CUDA devices (1.417s) 2022-12-01T10:05:58.7848766Z test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad) 2022-12-01T10:05:58.7849249Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1126 2022-12-01T10:05:58.7849646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1127 2022-12-01T10:05:58.7849974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1128 2022-12-01T10:05:58.7850298Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1129 2022-12-01T10:05:58.7850774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7851102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7851534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7851865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7852202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7852740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7853072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7853540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7853886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7854221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7854762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7855078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7855507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7855849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7856167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7856631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7856963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7857392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7857720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7858050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7858605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7859118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7859634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7860142Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7860530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7860864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7861212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7861550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7861878Z dist init r=1, world=4 2022-12-01T10:05:58.7862054Z dist init r=3, world=4 2022-12-01T10:05:58.7862235Z dist init r=2, world=4 2022-12-01T10:05:58.7862413Z dist init r=0, world=4 2022-12-01T10:05:58.7862607Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:05:58.7862877Z test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad) 2022-12-01T10:05:58.7863370Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1182 2022-12-01T10:05:58.7863786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1183 2022-12-01T10:05:58.7864110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1184 2022-12-01T10:05:58.7864423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1185 2022-12-01T10:05:58.7864884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7865395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7865880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7866229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7866554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7867028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7867362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7867787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7868120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7868457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7868928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7869254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7869667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7870011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7870342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7870825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7871320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7871649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7872071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7872402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7872734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7873225Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7873742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7874240Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7874623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7875021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7875355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7875698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7875952Z dist init r=0, world=4 2022-12-01T10:05:58.7876133Z dist init r=3, world=4 2022-12-01T10:05:58.7876298Z dist init r=2, world=4 2022-12-01T10:05:58.7876503Z dist init r=1, world=4 2022-12-01T10:05:58.7876707Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.7876972Z test_param_change_after_init_mixed_precision_False (__main__.TestParamInit) 2022-12-01T10:05:58.7877489Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1238 2022-12-01T10:05:58.7877892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1239 2022-12-01T10:05:58.7878328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1240 2022-12-01T10:05:58.7878659Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1241 2022-12-01T10:05:58.7879112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7879448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7879867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7880212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7880548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7881017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7881333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7881759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7882102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7882421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7882890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7883219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7883641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7883968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7884300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7884796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7885290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7885602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7886031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7886382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7886701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7887190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7887777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7888296Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7888672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7889024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7889407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7889752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7889998Z dist init r=3, world=4 2022-12-01T10:05:58.7890181Z dist init r=1, world=4 2022-12-01T10:05:58.7890358Z dist init r=0, world=4 2022-12-01T10:05:58.7890521Z dist init r=2, world=4 2022-12-01T10:05:58.7890725Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.7891007Z test_param_change_after_init_mixed_precision_True (__main__.TestParamInit) 2022-12-01T10:05:58.7891511Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1294 2022-12-01T10:05:58.7891919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1295 2022-12-01T10:05:58.7892243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1296 2022-12-01T10:05:58.7892567Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1297 2022-12-01T10:05:58.7893003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7893338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7893765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7894098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7894432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7894898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7895223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7895634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7895985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7896319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7896773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7897096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7897523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7897872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7898191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7898749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7899276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7899776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7900093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7900555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7900902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7901224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7901714Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7902221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7902636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7902969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7903313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7903655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7903914Z dist init r=1, world=4 2022-12-01T10:05:58.7904080Z dist init r=0, world=4 2022-12-01T10:05:58.7904258Z dist init r=3, world=4 2022-12-01T10:05:58.7904433Z dist init r=2, world=4 2022-12-01T10:05:58.7904626Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.7904907Z test_delayed_optim_step_offload_false_no_shard (__main__.TestParityWithDDP) 2022-12-01T10:05:58.7905305Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1350 2022-12-01T10:05:58.7905684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1351 2022-12-01T10:05:58.7906009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1352 2022-12-01T10:05:58.7906324Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1353 2022-12-01T10:05:58.7906782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7907098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7907527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7907874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7908196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7908667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7908993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7909418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7909745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7910083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7910550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7910873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7911288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7911637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7911970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7912418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7912743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7913198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7913541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7913860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7914356Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7914906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7915425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7915928Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7916312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7916662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7916995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7917338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7917593Z dist init r=3, world=4 2022-12-01T10:05:58.7917773Z dist init r=1, world=4 2022-12-01T10:05:58.7917940Z dist init r=0, world=4 2022-12-01T10:05:58.7918118Z dist init r=2, world=4 2022-12-01T10:05:58.7918639Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.7918900Z test_delayed_optim_step_offload_false_none (__main__.TestParityWithDDP) 2022-12-01T10:05:58.7919297Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1406 2022-12-01T10:05:58.7919690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1407 2022-12-01T10:05:58.7920011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1408 2022-12-01T10:05:58.7920335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1409 2022-12-01T10:05:58.7920786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7921117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7921539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7921886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7922223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7922692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7923032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7923459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7923800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7924121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7924587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7924917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7925344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7925673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7926085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7926554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7926865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7927293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7927635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7928009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7928490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7929008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7929529Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7930038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7930409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7930762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7931113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7931458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7931703Z dist init r=0, world=4 2022-12-01T10:05:58.7931884Z dist init r=1, world=4 2022-12-01T10:05:58.7932063Z dist init r=2, world=4 2022-12-01T10:05:58.7932228Z dist init r=3, world=4 2022-12-01T10:05:58.7932434Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.7932724Z test_delayed_optim_step_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-12-01T10:05:58.7933119Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1462 2022-12-01T10:05:58.7933510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1463 2022-12-01T10:05:58.7933836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1464 2022-12-01T10:05:58.7934161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1465 2022-12-01T10:05:58.7934597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7934927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7935356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7935690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7936023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7936492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7936817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7937233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7937582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7937916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7938426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7938756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7939215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7939573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7939909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7940393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7940946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7941275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7941705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7942034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7942372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7942867Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7943384Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7943888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7944280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7944634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7944979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7945318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7945576Z dist init r=1, world=4 2022-12-01T10:05:58.7945758Z dist init r=2, world=4 2022-12-01T10:05:58.7945926Z dist init r=3, world=4 2022-12-01T10:05:58.7946107Z dist init r=0, world=4 2022-12-01T10:05:58.7946315Z skip: Need at least 2 CUDA devices (1.416s) 2022-12-01T10:05:58.7946585Z test_delayed_optim_step_offload_true_no_shard (__main__.TestParityWithDDP) 2022-12-01T10:05:58.7947469Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82490 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-12-01T10:05:58.7948144Z test_delayed_optim_step_offload_true_none (__main__.TestParityWithDDP) 2022-12-01T10:05:58.7948543Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1518 2022-12-01T10:05:58.7948944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1519 2022-12-01T10:05:58.7949260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1520 2022-12-01T10:05:58.7949583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1521 2022-12-01T10:05:58.7950040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7950383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7950800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7951150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7951487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7951999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7952330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7952763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7953108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7953461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7953958Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7954462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7954795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7955213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7955560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7955897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7956356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7956686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7957114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7957458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7957779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7958424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7958963Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7959485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7959860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7960213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7960561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7960893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7961151Z dist init r=0, world=4 2022-12-01T10:05:58.7961336Z dist init r=3, world=4 2022-12-01T10:05:58.7961503Z dist init r=1, world=4 2022-12-01T10:05:58.7961679Z dist init r=2, world=4 2022-12-01T10:05:58.7961884Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.7962157Z test_delayed_optim_step_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-12-01T10:05:58.7962562Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1574 2022-12-01T10:05:58.7962954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1575 2022-12-01T10:05:58.7963282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1576 2022-12-01T10:05:58.7963591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1577 2022-12-01T10:05:58.7964040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7964371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7964863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7965200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7965538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7966010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7966365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7966796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7967146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7967486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7967946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7968271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7968704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7969049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7969374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7969851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7970303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7970759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7971173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7971574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7972134Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7972707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7973283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7973864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7974306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7974670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7975094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7975513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7975791Z dist init r=3, world=4 2022-12-01T10:05:58.7976029Z dist init r=2, world=4 2022-12-01T10:05:58.7976291Z dist init r=1, world=4 2022-12-01T10:05:58.7976488Z dist init r=0, world=4 2022-12-01T10:05:58.7976763Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.7977112Z test_delayed_reduce_scatter_offload_false_no_shard (__main__.TestParityWithDDP) 2022-12-01T10:05:58.7977580Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1630 2022-12-01T10:05:58.7978013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1631 2022-12-01T10:05:58.7978513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1632 2022-12-01T10:05:58.7978946Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1633 2022-12-01T10:05:58.7979440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7979837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7980353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7980793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7981146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.7981679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7982068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7982535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7982884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7983292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.7983818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7984185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7984668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7985075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7985464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.7985977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7986551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.7986971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.7987451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.7987810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.7988228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.7988787Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7989323Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7989895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.7990341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.7990776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.7991139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.7991549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.7991864Z dist init r=0, world=4 2022-12-01T10:05:58.8032986Z dist init r=1, world=4 2022-12-01T10:05:58.8033168Z dist init r=2, world=4 2022-12-01T10:05:58.8033338Z dist init r=3, world=4 2022-12-01T10:05:58.8033539Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8033807Z test_delayed_reduce_scatter_offload_false_none (__main__.TestParityWithDDP) 2022-12-01T10:05:58.8034841Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82704 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-12-01T10:05:58.8035442Z test_delayed_reduce_scatter_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-12-01T10:05:58.8036337Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82398 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-12-01T10:05:58.8036911Z test_delayed_reduce_scatter_offload_true_no_shard (__main__.TestParityWithDDP) 2022-12-01T10:05:58.8037308Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1686 2022-12-01T10:05:58.8037680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1687 2022-12-01T10:05:58.8037997Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1688 2022-12-01T10:05:58.8038478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1689 2022-12-01T10:05:58.8038922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8039239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8039654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8039986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8040307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8040762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8041077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8041483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8041814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8042134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8042585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8042891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8043304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8043637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8043960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8044437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8044919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8045239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8045653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8045988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8046307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8046834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8047346Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8047845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8048254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8048595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8048932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8049266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8049519Z dist init r=1, world=4 2022-12-01T10:05:58.8049689Z dist init r=2, world=4 2022-12-01T10:05:58.8049865Z dist init r=3, world=4 2022-12-01T10:05:58.8050040Z dist init r=0, world=4 2022-12-01T10:05:58.8050238Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8050522Z test_delayed_reduce_scatter_offload_true_none (__main__.TestParityWithDDP) 2022-12-01T10:05:58.8051386Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82399 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-12-01T10:05:58.8051991Z test_delayed_reduce_scatter_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-12-01T10:05:58.8052842Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82403 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-12-01T10:05:58.8053520Z test_mixture_of_experts_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1742 2022-12-01T10:05:58.8053932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1743 2022-12-01T10:05:58.8054258Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1744 2022-12-01T10:05:58.8054581Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1745 2022-12-01T10:05:58.8055021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8055351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8055782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8056117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8056435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8056902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8057236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8057652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8057996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8058395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8058929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8059436Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8059922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8060239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8060686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8061031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8061362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8061832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8062145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8062567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8062914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8063251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8063735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8064247Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8064631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8064968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8065313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8065654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8065917Z dist init r=2, world=4 2022-12-01T10:05:58.8066089Z dist init r=3, world=4 2022-12-01T10:05:58.8066264Z dist init r=1, world=4 2022-12-01T10:05:58.8066437Z dist init r=0, world=4 2022-12-01T10:05:58.8066630Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8067021Z test_mixture_of_experts_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1798 2022-12-01T10:05:58.8067432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1799 2022-12-01T10:05:58.8067756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1800 2022-12-01T10:05:58.8068061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1801 2022-12-01T10:05:58.8068511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8068845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8069261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8069603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8069935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8070398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8070704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8071119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8071482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8071805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8072265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8072596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8073018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8073373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8073692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8074151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8074479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8074890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8075228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8075559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8076038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8076566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8077073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8077575Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8077941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8078463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8078810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8079150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8079398Z dist init r=3, world=4 2022-12-01T10:05:58.8079631Z dist init r=1, world=4 2022-12-01T10:05:58.8079808Z dist init r=2, world=4 2022-12-01T10:05:58.8079976Z dist init r=0, world=4 2022-12-01T10:05:58.8080179Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8080564Z test_mixture_of_experts_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1854 2022-12-01T10:05:58.8080972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1855 2022-12-01T10:05:58.8081299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1856 2022-12-01T10:05:58.8081612Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1857 2022-12-01T10:05:58.8082056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8082373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8082797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8083135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8083462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8083975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8084298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8084725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8085064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8085394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8085894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8086222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8086635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8086977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8087310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8087761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8088088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8088515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8088857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8089177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8089671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8090193Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8090708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8091211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8091600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8091949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8092298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8092638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8092892Z dist init r=1, world=4 2022-12-01T10:05:58.8093073Z dist init r=2, world=4 2022-12-01T10:05:58.8093238Z dist init r=3, world=4 2022-12-01T10:05:58.8093414Z dist init r=0, world=4 2022-12-01T10:05:58.8093620Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8093991Z test_mixture_of_experts_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1910 2022-12-01T10:05:58.8094408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1911 2022-12-01T10:05:58.8094736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1912 2022-12-01T10:05:58.8095063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1913 2022-12-01T10:05:58.8095506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8095834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8096262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8096601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8096968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8097440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8097765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8098180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8098624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8098958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8099414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8099742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8100175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8100517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8100830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8101298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8101631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8102059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8102390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8102721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8103221Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8103733Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8104250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8104770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8105158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8105482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8105826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8106166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8106426Z dist init r=1, world=4 2022-12-01T10:05:58.8106593Z dist init r=2, world=4 2022-12-01T10:05:58.8106768Z dist init r=3, world=4 2022-12-01T10:05:58.8106944Z dist init r=0, world=4 2022-12-01T10:05:58.8107138Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8107518Z test_mixture_of_experts_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1966 2022-12-01T10:05:58.8107935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1967 2022-12-01T10:05:58.8108248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1968 2022-12-01T10:05:58.8108578Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1969 2022-12-01T10:05:58.8109032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8109398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8109810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8110152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8110487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8110953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8111303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8111731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8112072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8112395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8112892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8113384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8113712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8114124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8114472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8114805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8115284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8115779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8116108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8116536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8116866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8117199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8117694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8118352Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8118752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8119100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8119445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8119796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8120041Z dist init r=0, world=4 2022-12-01T10:05:58.8120222Z dist init r=1, world=4 2022-12-01T10:05:58.8120400Z dist init r=3, world=4 2022-12-01T10:05:58.8120562Z dist init r=2, world=4 2022-12-01T10:05:58.8120770Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8121167Z test_mixture_of_experts_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2022 2022-12-01T10:05:58.8121581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2023 2022-12-01T10:05:58.8121912Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2024 2022-12-01T10:05:58.8122314Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2025 2022-12-01T10:05:58.8122775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8123092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8123520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8123910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8124238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8124707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8125040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8125494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8125826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8126162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8126658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8127155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8127474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8127905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8128250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8128571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8129043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8129371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8129799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8130129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8130469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8130964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8131473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8131988Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8132375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8132722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8133056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8133397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8133659Z dist init r=3, world=4 2022-12-01T10:05:58.8133839Z dist init r=1, world=4 2022-12-01T10:05:58.8134002Z dist init r=2, world=4 2022-12-01T10:05:58.8134182Z dist init r=0, world=4 2022-12-01T10:05:58.8134386Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8134772Z test_mixture_of_experts_with_delay_before_free_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2078 2022-12-01T10:05:58.8135228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2079 2022-12-01T10:05:58.8135557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2080 2022-12-01T10:05:58.8135870Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2081 2022-12-01T10:05:58.8136323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8136684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8137115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8137449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8137784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8138257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8138649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8139062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8139411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8139745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8140206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8140537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8140964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8141308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8141630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8142120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8142616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8142929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8143359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8143704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8144036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8144517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8145033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8145543Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8145926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8146263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8146607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8146951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8147198Z dist init r=2, world=4 2022-12-01T10:05:58.8147377Z dist init r=1, world=4 2022-12-01T10:05:58.8147557Z dist init r=0, world=4 2022-12-01T10:05:58.8147719Z dist init r=3, world=4 2022-12-01T10:05:58.8147962Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8148358Z test_mixture_of_experts_with_delay_before_free_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2134 2022-12-01T10:05:58.8148785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2135 2022-12-01T10:05:58.8149107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2136 2022-12-01T10:05:58.8149458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2137 2022-12-01T10:05:58.8149906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8150242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8150656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8151007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8151340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8151794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8152128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8152551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8152897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8153217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8153685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8154012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8154426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8154765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8155100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8155590Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8156072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8156397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8156819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8157168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8157491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8157979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8158660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8159177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8159552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8159903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8160248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8160634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8160893Z dist init r=3, world=4 2022-12-01T10:05:58.8161074Z dist init r=2, world=4 2022-12-01T10:05:58.8161236Z dist init r=0, world=4 2022-12-01T10:05:58.8161410Z dist init r=1, world=4 2022-12-01T10:05:58.8161616Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8162018Z test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2190 2022-12-01T10:05:58.8162470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2191 2022-12-01T10:05:58.8162796Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2192 2022-12-01T10:05:58.8163120Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2193 2022-12-01T10:05:58.8163563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8163893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8164319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8164663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8164985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8165451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8165781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8166207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8166538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8166875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8167339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8167652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8168073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8168417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8168751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8169201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8169527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8169953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8170280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8170610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8171101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8171619Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8172120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8172631Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8173019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8173394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8173559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8173710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8173785Z dist init r=2, world=4 2022-12-01T10:05:58.8173858Z dist init r=0, world=4 2022-12-01T10:05:58.8173962Z dist init r=3, world=4 2022-12-01T10:05:58.8174037Z dist init r=1, world=4 2022-12-01T10:05:58.8174141Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8174402Z test_mixture_of_experts_with_delay_before_free_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2246 2022-12-01T10:05:58.8174545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2247 2022-12-01T10:05:58.8174699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2248 2022-12-01T10:05:58.8174850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2249 2022-12-01T10:05:58.8175129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8175253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8175535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8175675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8175850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8176121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8176231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8176514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8176650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8176828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8177101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8177226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8177504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8177638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8177800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8178072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8178193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8178542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8178678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8178856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8179161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8179459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8179749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8180060Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8180228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8180391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8180553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8180745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8180821Z dist init r=1, world=4 2022-12-01T10:05:58.8180895Z dist init r=0, world=4 2022-12-01T10:05:58.8180971Z dist init r=3, world=4 2022-12-01T10:05:58.8181031Z dist init r=2, world=4 2022-12-01T10:05:58.8181134Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8181398Z test_mixture_of_experts_with_delay_before_free_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2302 2022-12-01T10:05:58.8181553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2303 2022-12-01T10:05:58.8181704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2304 2022-12-01T10:05:58.8181855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2305 2022-12-01T10:05:58.8182134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8182261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8182532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8182666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8182841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8183114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8183236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8183514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8183649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8183826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8184097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8184208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8184487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8184623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8184795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8185095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8185364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8185488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8185765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8185898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8186061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8186386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8186687Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8186976Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8187141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8187332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8187495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8187657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8187719Z dist init r=2, world=4 2022-12-01T10:05:58.8187792Z dist init r=3, world=4 2022-12-01T10:05:58.8187867Z dist init r=0, world=4 2022-12-01T10:05:58.8187940Z dist init r=1, world=4 2022-12-01T10:05:58.8188044Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8188315Z test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2358 2022-12-01T10:05:58.8188469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2359 2022-12-01T10:05:58.8188623Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2360 2022-12-01T10:05:58.8188763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2361 2022-12-01T10:05:58.8189036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8189159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8189445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8189582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8189756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8190027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8190149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8190430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8190552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8190726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8191024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8191294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8191417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8191695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8191829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8192005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8192262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8192385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8192707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8192847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8193021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8193319Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8193614Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8193935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8194098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8194249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8194412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8194573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8194648Z dist init r=3, world=4 2022-12-01T10:05:58.8194720Z dist init r=2, world=4 2022-12-01T10:05:58.8194792Z dist init r=1, world=4 2022-12-01T10:05:58.8194863Z dist init r=0, world=4 2022-12-01T10:05:58.8194954Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8195218Z test_nested_always_wrap_model_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2414 2022-12-01T10:05:58.8195374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2415 2022-12-01T10:05:58.8195528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2416 2022-12-01T10:05:58.8195679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2417 2022-12-01T10:05:58.8195956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8196078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8196358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8196494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8196657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8196930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8197052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8197331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8197469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8197645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8197914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8198037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8198485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8198612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8198786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8199062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8199184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8199524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8199662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8199838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8200137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8200473Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8200750Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8201038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8201206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8201370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8201533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8201696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8201771Z dist init r=3, world=4 2022-12-01T10:05:58.8201849Z dist init r=0, world=4 2022-12-01T10:05:58.8201909Z dist init r=2, world=4 2022-12-01T10:05:58.8201985Z dist init r=1, world=4 2022-12-01T10:05:58.8202088Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8202345Z test_nested_always_wrap_model_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2470 2022-12-01T10:05:58.8202500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2471 2022-12-01T10:05:58.8202655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2472 2022-12-01T10:05:58.8202807Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2473 2022-12-01T10:05:58.8203083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8203194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8203478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8203614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8203792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8204062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8204186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8204465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8204601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8204765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8205034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8205159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8205440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8205572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8205775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8206046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8206167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8206446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8206569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8206772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8207069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8207365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8207657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8207946Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8208111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8208276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8208442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8208595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8208672Z dist init r=0, world=4 2022-12-01T10:05:58.8208747Z dist init r=2, world=4 2022-12-01T10:05:58.8208819Z dist init r=3, world=4 2022-12-01T10:05:58.8208891Z dist init r=1, world=4 2022-12-01T10:05:58.8208993Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8209263Z test_nested_always_wrap_model_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2526 2022-12-01T10:05:58.8209419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2527 2022-12-01T10:05:58.8209559Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2528 2022-12-01T10:05:58.8209711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2529 2022-12-01T10:05:58.8209989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8210111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8210391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8210525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8210702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8210979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8211088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8211366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8211504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8211679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8211975Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8212243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8212394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8212672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8212805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8212969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8213267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8213389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8213665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8213797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8213978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8214274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8214568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8214860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8215017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8215180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8215345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8215506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8215584Z dist init r=1, world=4 2022-12-01T10:05:58.8215659Z dist init r=2, world=4 2022-12-01T10:05:58.8215734Z dist init r=0, world=4 2022-12-01T10:05:58.8215794Z dist init r=3, world=4 2022-12-01T10:05:58.8215898Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8216158Z test_nested_always_wrap_model_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2582 2022-12-01T10:05:58.8216314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2583 2022-12-01T10:05:58.8216468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2584 2022-12-01T10:05:58.8216618Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2585 2022-12-01T10:05:58.8216892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8217015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8217299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8217421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8217593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8217862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8217988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8218266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8218463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8218639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8218944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8219067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8219333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8219465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8219668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8219939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8220062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8220339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8220475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8220648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8220931Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8221229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8221521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8221812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8221980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8222144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8222309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8222470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8222545Z dist init r=2, world=4 2022-12-01T10:05:58.8222606Z dist init r=0, world=4 2022-12-01T10:05:58.8222678Z dist init r=3, world=4 2022-12-01T10:05:58.8222749Z dist init r=1, world=4 2022-12-01T10:05:58.8222856Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8223112Z test_nested_always_wrap_model_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2638 2022-12-01T10:05:58.8223266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2639 2022-12-01T10:05:58.8223419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2640 2022-12-01T10:05:58.8223575Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2641 2022-12-01T10:05:58.8223840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8223963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8224245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8224381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8224559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8224828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8224951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8225234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8225387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8225565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8225835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8225960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8226268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8226401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8226577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8226877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8227151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8227262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8227538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8227669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8227848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8228144Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8228441Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8228734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8228900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8229065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8229216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8229375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8229452Z dist init r=0, world=4 2022-12-01T10:05:58.8229525Z dist init r=3, world=4 2022-12-01T10:05:58.8229598Z dist init r=1, world=4 2022-12-01T10:05:58.8229669Z dist init r=2, world=4 2022-12-01T10:05:58.8229772Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:05:58.8230024Z test_nested_always_wrap_model_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2694 2022-12-01T10:05:58.8230182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2695 2022-12-01T10:05:58.8230333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2696 2022-12-01T10:05:58.8230482Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2697 2022-12-01T10:05:58.8230758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8230883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8231164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8231298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8231474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8231763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8231887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8232169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8232306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8232482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8232781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8232906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8233187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8233309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8233487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8233756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8233879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8234157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8234292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8234466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8234764Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8235060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8235341Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8235633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8235800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8235964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8236127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8236288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8236363Z dist init r=1, world=4 2022-12-01T10:05:58.8236435Z dist init r=2, world=4 2022-12-01T10:05:58.8236495Z dist init r=0, world=4 2022-12-01T10:05:58.8236566Z dist init r=3, world=4 2022-12-01T10:05:58.8236671Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8236918Z test_nested_wrapped_model_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2750 2022-12-01T10:05:58.8237072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2751 2022-12-01T10:05:58.8237224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2752 2022-12-01T10:05:58.8237378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2753 2022-12-01T10:05:58.8237653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8237776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8238046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8238417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8238602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8238880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8239004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8239285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8239458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8239635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8239893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8240015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8240294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8240430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8240604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8240875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8240999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8241277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8241410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8241573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8241870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8242167Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8242457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8242745Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8242913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8243077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8243238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8243404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8243467Z dist init r=3, world=4 2022-12-01T10:05:58.8243541Z dist init r=1, world=4 2022-12-01T10:05:58.8243612Z dist init r=2, world=4 2022-12-01T10:05:58.8243685Z dist init r=0, world=4 2022-12-01T10:05:58.8243789Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8244030Z test_nested_wrapped_model_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2806 2022-12-01T10:05:58.8244192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2807 2022-12-01T10:05:58.8244333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2808 2022-12-01T10:05:58.8244485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2809 2022-12-01T10:05:58.8244760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8244911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8245195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8245330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8245505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8245775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8245927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8246195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8246329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8246503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8246773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8246896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8247176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8247311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8247488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8247772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8248070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8248343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8248465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8248742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8248876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8249051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8249348Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8249639Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8249792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8249949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8250111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8250272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8250344Z dist init r=0, world=4 2022-12-01T10:05:58.8250419Z dist init r=1, world=4 2022-12-01T10:05:58.8250492Z dist init r=2, world=4 2022-12-01T10:05:58.8250564Z dist init r=3, world=4 2022-12-01T10:05:58.8250662Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8250911Z test_nested_wrapped_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2862 2022-12-01T10:05:58.8251069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2863 2022-12-01T10:05:58.8251221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2864 2022-12-01T10:05:58.8251407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2865 2022-12-01T10:05:58.8251681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8251803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8252087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8252239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8252415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8252690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8252815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8253097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8253232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8253407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8253678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8253799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8254067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8254202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8254377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8254645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8254769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8255046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8255178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8255353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8255654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8255939Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8256234Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8256527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8256692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8256854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8257018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8257180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8257257Z dist init r=3, world=4 2022-12-01T10:05:58.8257317Z dist init r=0, world=4 2022-12-01T10:05:58.8257389Z dist init r=1, world=4 2022-12-01T10:05:58.8257461Z dist init r=2, world=4 2022-12-01T10:05:58.8257564Z skip: Need at least 2 CUDA devices (1.416s) 2022-12-01T10:05:58.8257808Z test_nested_wrapped_model_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2918 2022-12-01T10:05:58.8257992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2919 2022-12-01T10:05:58.8258149Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2920 2022-12-01T10:05:58.8258299Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2921 2022-12-01T10:05:58.8258623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8258778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8259061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8259199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8259375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8259648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8259772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8260052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8260185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8260348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8260623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8260747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8261026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8261160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8261334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8261606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8261730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8261997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8262134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8262309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8262604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8262902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8263200Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8263489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8263653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8263810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8263961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8264123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8264198Z dist init r=0, world=4 2022-12-01T10:05:58.8264273Z dist init r=1, world=4 2022-12-01T10:05:58.8264344Z dist init r=2, world=4 2022-12-01T10:05:58.8264416Z dist init r=3, world=4 2022-12-01T10:05:58.8264548Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8264791Z test_nested_wrapped_model_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2974 2022-12-01T10:05:58.8264933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2975 2022-12-01T10:05:58.8265085Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2976 2022-12-01T10:05:58.8265265Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2977 2022-12-01T10:05:58.8265543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8265667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8265949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8266085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8266261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8266523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8266646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8266925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8267060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8267237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8267510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8267634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8267916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8268051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8268213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8268484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8268609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8268886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8269019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8269194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8269498Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8269792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8270088Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8270366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8270533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8270693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8270855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8271042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8271118Z dist init r=3, world=4 2022-12-01T10:05:58.8271195Z dist init r=0, world=4 2022-12-01T10:05:58.8271270Z dist init r=2, world=4 2022-12-01T10:05:58.8271331Z dist init r=1, world=4 2022-12-01T10:05:58.8271433Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8271682Z test_nested_wrapped_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3030 2022-12-01T10:05:58.8271864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3031 2022-12-01T10:05:58.8272015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3032 2022-12-01T10:05:58.8272165Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3033 2022-12-01T10:05:58.8272439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8272563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8272832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8272967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8273142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8273413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8273540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8273823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8273957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8274133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8274391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8274517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8274796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8274931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8275108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8275376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8275499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8275776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8275911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8276074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8276370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8276666Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8276965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8277256Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8277421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8277635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8277798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8277961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8278024Z dist init r=2, world=4 2022-12-01T10:05:58.8278099Z dist init r=0, world=4 2022-12-01T10:05:58.8278172Z dist init r=1, world=4 2022-12-01T10:05:58.8278390Z dist init r=3, world=4 2022-12-01T10:05:58.8278554Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8278835Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3086 2022-12-01T10:05:58.8278990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3087 2022-12-01T10:05:58.8279141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3088 2022-12-01T10:05:58.8279282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3089 2022-12-01T10:05:58.8279562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8279686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8279970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8280107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8280282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8280551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8280674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8280944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8281079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8281254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8281524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8281646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8281927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8282060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8282234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8282535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8282793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8282913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8283192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8283324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8283499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8283797Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8284093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8284436Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8284603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8284751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8284913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8285101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8285176Z dist init r=0, world=4 2022-12-01T10:05:58.8285249Z dist init r=2, world=4 2022-12-01T10:05:58.8285320Z dist init r=3, world=4 2022-12-01T10:05:58.8285391Z dist init r=1, world=4 2022-12-01T10:05:58.8285482Z skip: Need at least 2 CUDA devices (1.515s) 2022-12-01T10:05:58.8285763Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3142 2022-12-01T10:05:58.8285917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3143 2022-12-01T10:05:58.8286069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3144 2022-12-01T10:05:58.8286219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3145 2022-12-01T10:05:58.8286495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8286624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8286903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8287040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8287203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8287480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8287603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8287881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8288016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8288193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8288462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8288583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8288848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8288984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8289156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8289428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8289549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8289826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8289960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8290137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8290437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8290746Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8291042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8291330Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8291494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8291682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8291846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8292006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8292084Z dist init r=2, world=4 2022-12-01T10:05:58.8292158Z dist init r=0, world=4 2022-12-01T10:05:58.8292218Z dist init r=3, world=4 2022-12-01T10:05:58.8292293Z dist init r=1, world=4 2022-12-01T10:05:58.8292398Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8292682Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3198 2022-12-01T10:05:58.8292838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3199 2022-12-01T10:05:58.8292992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3200 2022-12-01T10:05:58.8293145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3201 2022-12-01T10:05:58.8293425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8293537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8293821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8293958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8294135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8294406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8294527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8294810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8294942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8295107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8295378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8295501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8295783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8295915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8296090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8296389Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8296657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8296778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8297043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8297198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8297371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8297665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8297957Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8298275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8298500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8298666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8298827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8298981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8299056Z dist init r=2, world=4 2022-12-01T10:05:58.8299129Z dist init r=1, world=4 2022-12-01T10:05:58.8299202Z dist init r=0, world=4 2022-12-01T10:05:58.8299274Z dist init r=3, world=4 2022-12-01T10:05:58.8299377Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8299654Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3254 2022-12-01T10:05:58.8299814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3255 2022-12-01T10:05:58.8299953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3256 2022-12-01T10:05:58.8300104Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3257 2022-12-01T10:05:58.8300386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8300508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8300788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8300922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8301095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8301370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8301479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8301763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8301899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8302075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8302344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8302465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8302740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8302875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8303047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8303305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8303424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8303732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8303867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8304041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8304336Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8304665Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8304957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8305247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8305402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8305566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8305728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8305889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8305964Z dist init r=2, world=4 2022-12-01T10:05:58.8306038Z dist init r=0, world=4 2022-12-01T10:05:58.8306109Z dist init r=1, world=4 2022-12-01T10:05:58.8306168Z dist init r=3, world=4 2022-12-01T10:05:58.8306270Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8306543Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3310 2022-12-01T10:05:58.8306698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3311 2022-12-01T10:05:58.8306851Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3312 2022-12-01T10:05:58.8307002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3313 2022-12-01T10:05:58.8307276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8307398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8307683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8307807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8307981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8308252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8308377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8308657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8308791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8308968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8309235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8309347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8309622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8309755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8309956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8310252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8310520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8310641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8310917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8311079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8311242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8311532Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8311829Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8312120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8312284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8312447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8312611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8312771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8312844Z dist init r=0, world=4 2022-12-01T10:05:58.8312906Z dist init r=2, world=4 2022-12-01T10:05:58.8312984Z dist init r=1, world=4 2022-12-01T10:05:58.8313054Z dist init r=3, world=4 2022-12-01T10:05:58.8313155Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8313439Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3366 2022-12-01T10:05:58.8313592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3367 2022-12-01T10:05:58.8313743Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3368 2022-12-01T10:05:58.8313881Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3369 2022-12-01T10:05:58.8314158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8314280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8314557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8314691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8314870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8315142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8315263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8315542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8315666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8315836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8316103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8316223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8316528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8316665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8316836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8317133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8317430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8317539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8317814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8317945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8318116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8318613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8318914Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8319204Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8319371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8319534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8319684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8319842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8319917Z dist init r=0, world=4 2022-12-01T10:05:58.8319988Z dist init r=1, world=4 2022-12-01T10:05:58.8320059Z dist init r=2, world=4 2022-12-01T10:05:58.8320129Z dist init r=3, world=4 2022-12-01T10:05:58.8320231Z skip: Need at least 2 CUDA devices (1.515s) 2022-12-01T10:05:58.8320468Z test_transformer_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3422 2022-12-01T10:05:58.8320625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3423 2022-12-01T10:05:58.8320779Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3424 2022-12-01T10:05:58.8320929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3425 2022-12-01T10:05:58.8321200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8321322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8321604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8321740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8321913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8322171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8322288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8322566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8322698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8322870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8323188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8323311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8323586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8323707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8323920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8324191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8324314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8324587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8324721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8324894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8325191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8325487Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8325766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8326053Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8326218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8326380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8326545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8326705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8326780Z dist init r=2, world=4 2022-12-01T10:05:58.8326853Z dist init r=0, world=4 2022-12-01T10:05:58.8326912Z dist init r=1, world=4 2022-12-01T10:05:58.8326983Z dist init r=3, world=4 2022-12-01T10:05:58.8327089Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8327335Z test_transformer_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3478 2022-12-01T10:05:58.8327491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3479 2022-12-01T10:05:58.8327641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3480 2022-12-01T10:05:58.8327791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3481 2022-12-01T10:05:58.8328064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8328174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8328452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8328581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8328758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8329027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8329152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8329425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8329599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8329774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8330031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8330153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8330461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8330592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8330766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8331032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8331154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8331428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8331564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8331724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8332017Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8332310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8332592Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8332876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8333045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8333205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8333366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8333523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8333588Z dist init r=3, world=4 2022-12-01T10:05:58.8333657Z dist init r=0, world=4 2022-12-01T10:05:58.8333727Z dist init r=2, world=4 2022-12-01T10:05:58.8333799Z dist init r=1, world=4 2022-12-01T10:05:58.8333903Z skip: Need at least 2 CUDA devices (1.415s) 2022-12-01T10:05:58.8334159Z test_transformer_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3534 2022-12-01T10:05:58.8334316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3535 2022-12-01T10:05:58.8334456Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3536 2022-12-01T10:05:58.8334607Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3537 2022-12-01T10:05:58.8334880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8335006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8335290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8335426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8335601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8335898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8336022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8336289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8336423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8336599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8336920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8337042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8337322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8337457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8337635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8337922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8338218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8338555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8338679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8338961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8339095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8339268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8339566Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8339861Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8340014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8340174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8340338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8340496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8340571Z dist init r=1, world=4 2022-12-01T10:05:58.8340643Z dist init r=0, world=4 2022-12-01T10:05:58.8340714Z dist init r=3, world=4 2022-12-01T10:05:58.8340785Z dist init r=2, world=4 2022-12-01T10:05:58.8340880Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:05:58.8341128Z test_transformer_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3590 2022-12-01T10:05:58.8341280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3591 2022-12-01T10:05:58.8341432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3592 2022-12-01T10:05:58.8341583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3593 2022-12-01T10:05:58.8341860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8341982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8342262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8342383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8342586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8342859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8342982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8343260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8343420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8343597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8343863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8343983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8344253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8344387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8344558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8344825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8344949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8345225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8345356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8345530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8345817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8346111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8346402Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8346689Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8346855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8347011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8347174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8347331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8347409Z dist init r=1, world=4 2022-12-01T10:05:58.8347470Z dist init r=0, world=4 2022-12-01T10:05:58.8347540Z dist init r=2, world=4 2022-12-01T10:05:58.8347611Z dist init r=3, world=4 2022-12-01T10:05:58.8347715Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8347963Z test_transformer_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3646 2022-12-01T10:05:58.8348118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3647 2022-12-01T10:05:58.8348271Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3648 2022-12-01T10:05:58.8348422Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3649 2022-12-01T10:05:58.8348684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8348831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8349110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8349245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8349419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8349686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8349832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8350104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8350225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8350399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8350668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8350789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8351066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8351199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8351376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8351636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8351752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8352016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8352145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8352311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8352605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8352896Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8353186Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8353470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8353631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8353787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8353936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8354092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8354164Z dist init r=3, world=4 2022-12-01T10:05:58.8354231Z dist init r=0, world=4 2022-12-01T10:05:58.8354296Z dist init r=1, world=4 2022-12-01T10:05:58.8354361Z dist init r=2, world=4 2022-12-01T10:05:58.8354462Z skip: Need at least 2 CUDA devices (1.315s) 2022-12-01T10:05:58.8354707Z test_transformer_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3702 2022-12-01T10:05:58.8354848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3703 2022-12-01T10:05:58.8354996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3704 2022-12-01T10:05:58.8355170Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3705 2022-12-01T10:05:58.8355437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8355554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8355831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8355961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8356159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:05:58.8356419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8356538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8356814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8356941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8357111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:05:58.8357372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8357488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8357767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8357902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8358068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:05:58.8358526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8358822Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8359090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:05:58.8359209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:05:58.8359482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:05:58.8359613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:05:58.8359788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:05:58.8360079Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8360366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:05:58.8360528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:05:58.8360685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:05:58.8360845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:05:58.8360998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:05:58.8361069Z dist init r=1, world=4 2022-12-01T10:05:58.8361140Z dist init r=0, world=4 2022-12-01T10:05:58.8361208Z dist init r=3, world=4 2022-12-01T10:05:58.8361268Z dist init r=2, world=4 2022-12-01T10:05:58.8361368Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:05:58.8361376Z 2022-12-01T10:05:58.8361575Z ---------------------------------------------------------------------- 2022-12-01T10:05:58.8361648Z Ran 59 tests in 73.352s 2022-12-01T10:05:58.8361653Z 2022-12-01T10:05:58.8361773Z OK (skipped=59) 2022-12-01T10:05:58.8361779Z 2022-12-01T10:05:58.8361862Z Generating XML reports... 2022-12-01T10:05:58.8362168Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20221201100445.xml 2022-12-01T10:05:58.8362472Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20221201100445.xml 2022-12-01T10:05:58.8362774Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20221201100445.xml 2022-12-01T10:05:58.8363126Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20221201100445.xml 2022-12-01T10:05:58.8363132Z 2022-12-01T10:05:58.8363505Z ##[endgroup] 2022-12-01T10:05:58.8363843Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_core_03u0b8q1) 2022-12-01T10:05:58.8363849Z 2022-12-01T10:05:58.8364061Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-12-01 10:05:58.772566] 2022-12-01T10:05:58.8364435Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:05:58.772925] 2022-12-01T10:07:27.8122222Z 2022-12-01T10:07:27.8173143Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params 2022-12-01T10:07:27.8174395Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_4sdjf84n) 2022-12-01T10:07:27.8175492Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params 2022-12-01T10:07:27.8175863Z 2022-12-01T10:07:27.8175992Z Running tests... 2022-12-01T10:07:27.8176498Z ---------------------------------------------------------------------- 2022-12-01T10:07:27.8177188Z test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3769 2022-12-01T10:07:27.8177930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3770 2022-12-01T10:07:27.8178744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8179293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8180018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8180599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8181162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8181935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8182456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8183163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8183732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8184272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8185203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8186062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8186448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8186784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8187218Z dist init r=0, world=2 2022-12-01T10:07:27.8187404Z dist init r=1, world=2 2022-12-01T10:07:27.8187601Z skip: Need at least 2 CUDA devices (1.431s) 2022-12-01T10:07:27.8187975Z test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3797 2022-12-01T10:07:27.8188378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3798 2022-12-01T10:07:27.8188897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8189222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8189652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8190493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8190837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8191291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8191619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8192043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8192385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8192708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8193204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8193722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8194099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8194447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8194702Z dist init r=0, world=2 2022-12-01T10:07:27.8194882Z dist init r=1, world=2 2022-12-01T10:07:27.8195076Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8195369Z test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams) 2022-12-01T10:07:27.8195776Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3825 2022-12-01T10:07:27.8196148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3826 2022-12-01T10:07:27.8196598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8196927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8197362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8197693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8198029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8198666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8198986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8199414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8199760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8200099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8200660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8201180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8201568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8201917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8202242Z dist init r=1, world=2 2022-12-01T10:07:27.8202427Z dist init r=0, world=2 2022-12-01T10:07:27.8202637Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8202917Z test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams) 2022-12-01T10:07:27.8203322Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3853 2022-12-01T10:07:27.8203717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3854 2022-12-01T10:07:27.8204170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8204489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8204918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8205268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8205593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8206062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8206391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8206820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8207149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8207486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8207976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8208481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8208871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8209222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8209481Z dist init r=0, world=2 2022-12-01T10:07:27.8209650Z dist init r=1, world=2 2022-12-01T10:07:27.8209858Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8210165Z test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams) 2022-12-01T10:07:27.8210565Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3881 2022-12-01T10:07:27.8210952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3882 2022-12-01T10:07:27.8211403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8211734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8212151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8212491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8212826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8213325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8213637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8214062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8214402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8214719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8215243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8215761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8216143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8216480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8216736Z dist init r=1, world=2 2022-12-01T10:07:27.8216918Z dist init r=0, world=2 2022-12-01T10:07:27.8217114Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8217416Z test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams) 2022-12-01T10:07:27.8217834Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3909 2022-12-01T10:07:27.8218291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3910 2022-12-01T10:07:27.8218729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8219057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8219492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8219826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8220164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8220633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8220961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8221382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8221722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8222054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8222553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8223062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8223446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8223792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8224039Z dist init r=0, world=2 2022-12-01T10:07:27.8224222Z dist init r=1, world=2 2022-12-01T10:07:27.8265428Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8265881Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3937 2022-12-01T10:07:27.8266340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3938 2022-12-01T10:07:27.8267004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8267344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8267786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8268120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8268463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8268994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8269323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8269735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8270085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8270426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8270913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8271434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8271827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8272181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8272433Z dist init r=0, world=2 2022-12-01T10:07:27.8272614Z dist init r=1, world=2 2022-12-01T10:07:27.8272823Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8273230Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3965 2022-12-01T10:07:27.8273680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3966 2022-12-01T10:07:27.8274132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8274458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8274874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8275219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8275551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8276013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8276326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8276754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8277096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8277414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8277903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8278597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8278990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8279327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8279585Z dist init r=1, world=2 2022-12-01T10:07:27.8279768Z dist init r=0, world=2 2022-12-01T10:07:27.8280014Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8280435Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3993 2022-12-01T10:07:27.8280884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3994 2022-12-01T10:07:27.8281334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8281693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8282123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8282469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8282802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8283259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8283587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8284012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8284340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8284675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8285164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8285676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8286053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8286401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8286656Z dist init r=0, world=2 2022-12-01T10:07:27.8286821Z dist init r=1, world=2 2022-12-01T10:07:27.8287030Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8287442Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4021 2022-12-01T10:07:27.8287893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4022 2022-12-01T10:07:27.8288329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8288656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8289083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8289425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8289743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8290211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8290535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8290949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8291291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8291622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8292112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8292648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8293032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8293375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8293631Z dist init r=1, world=2 2022-12-01T10:07:27.8293798Z dist init r=0, world=2 2022-12-01T10:07:27.8294035Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8294450Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4049 2022-12-01T10:07:27.8294886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4050 2022-12-01T10:07:27.8295339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8295665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8296092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8296419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8296756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8297223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8297533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8298027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8298375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8298713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8299195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8299706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8300090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8300441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8300688Z dist init r=1, world=2 2022-12-01T10:07:27.8300872Z dist init r=0, world=2 2022-12-01T10:07:27.8301081Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8301486Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4077 2022-12-01T10:07:27.8301931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4078 2022-12-01T10:07:27.8302377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8302704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8303118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8303469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8303800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8304254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8304576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8305036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8305377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8305697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8306189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8306732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8307115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8307449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8307700Z dist init r=0, world=2 2022-12-01T10:07:27.8307878Z dist init r=1, world=2 2022-12-01T10:07:27.8308074Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8308485Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4105 2022-12-01T10:07:27.8308930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4106 2022-12-01T10:07:27.8309375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8309692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8310117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8310463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8310781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8311246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8311571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8311996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8312325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8312662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8313153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8313669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8314043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8314391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8314648Z dist init r=0, world=2 2022-12-01T10:07:27.8314816Z dist init r=1, world=2 2022-12-01T10:07:27.8315021Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8315430Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4133 2022-12-01T10:07:27.8315873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4134 2022-12-01T10:07:27.8316306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8316630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8317055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8317417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8317753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8318364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8318700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8319176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8319523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8319860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8320343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8320868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8321257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8321610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8321854Z dist init r=1, world=2 2022-12-01T10:07:27.8322034Z dist init r=0, world=2 2022-12-01T10:07:27.8322244Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8322645Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4161 2022-12-01T10:07:27.8323090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4162 2022-12-01T10:07:27.8323542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8323869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8324285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8324632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8324964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8325433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8325744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8326173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8326518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8326839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8327333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8327851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8328233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8328570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8328818Z dist init r=0, world=2 2022-12-01T10:07:27.8328997Z dist init r=1, world=2 2022-12-01T10:07:27.8329188Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8329602Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4189 2022-12-01T10:07:27.8330087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4190 2022-12-01T10:07:27.8330540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8330856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8331286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8331683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8332022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8332477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8332807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8333234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8333565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8333900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8334397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8334921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8335297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8335641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8335898Z dist init r=0, world=2 2022-12-01T10:07:27.8336081Z dist init r=1, world=2 2022-12-01T10:07:27.8336276Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8336690Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4217 2022-12-01T10:07:27.8337133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4218 2022-12-01T10:07:27.8337573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8337961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8338399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8338749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8339069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8339542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8339875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8340293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8340641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8340975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8341472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8341978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8342365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8342743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8343002Z dist init r=1, world=2 2022-12-01T10:07:27.8343168Z dist init r=0, world=2 2022-12-01T10:07:27.8343376Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8343792Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4245 2022-12-01T10:07:27.8344248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4246 2022-12-01T10:07:27.8344700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8345031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8345460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8345792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8346127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8346594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8346906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8347329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8347674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8348004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8348484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8349000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8349384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8349729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8349974Z dist init r=1, world=2 2022-12-01T10:07:27.8350151Z dist init r=0, world=2 2022-12-01T10:07:27.8350354Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8350756Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4273 2022-12-01T10:07:27.8351196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4274 2022-12-01T10:07:27.8351647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8351978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8352394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8352737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8353068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8353522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8353851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8354278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8354624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8354972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8355466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8355985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8356367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8356732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8356987Z dist init r=0, world=2 2022-12-01T10:07:27.8357165Z dist init r=1, world=2 2022-12-01T10:07:27.8357359Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8357772Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4301 2022-12-01T10:07:27.8358357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4302 2022-12-01T10:07:27.8358849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8359171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8359606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8359958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8360282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8360754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8361084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8361514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8361844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8362178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8362671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8363190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8363564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8363909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8364202Z dist init r=1, world=2 2022-12-01T10:07:27.8364368Z dist init r=0, world=2 2022-12-01T10:07:27.8364578Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8364988Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4329 2022-12-01T10:07:27.8365428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4330 2022-12-01T10:07:27.8365865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8366195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8366620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8366950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8367282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8367810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8368141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8368556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8368903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8369242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8369787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8370290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8370674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8371025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8371271Z dist init r=1, world=2 2022-12-01T10:07:27.8371451Z dist init r=0, world=2 2022-12-01T10:07:27.8371655Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8372051Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4357 2022-12-01T10:07:27.8372491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4358 2022-12-01T10:07:27.8372941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8373268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8373686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8374029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8374361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8374829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8375138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8375561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8375906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8376224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8376718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8377234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8377620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8378015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8378282Z dist init r=0, world=2 2022-12-01T10:07:27.8378467Z dist init r=1, world=2 2022-12-01T10:07:27.8378665Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8378936Z test_raises_rank0_with_writeback (__main__.TestSummonFullParams) 2022-12-01T10:07:27.8379319Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4385 2022-12-01T10:07:27.8379704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4386 2022-12-01T10:07:27.8380146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8380511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8380943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8381290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8381611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8382076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8382431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8382846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8383189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8383529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8384028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8384535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8384923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8385270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8385529Z dist init r=1, world=2 2022-12-01T10:07:27.8385696Z dist init r=0, world=2 2022-12-01T10:07:27.8385904Z skip: Need at least 2 CUDA devices (1.209s) 2022-12-01T10:07:27.8386341Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4413 2022-12-01T10:07:27.8386816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4414 2022-12-01T10:07:27.8387270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8387597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8388023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8388357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8388689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8389155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8389465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8389893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8390239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8390576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8391053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8391573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8391958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8392303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8392551Z dist init r=0, world=2 2022-12-01T10:07:27.8392731Z dist init r=1, world=2 2022-12-01T10:07:27.8392940Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8393404Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4441 2022-12-01T10:07:27.8393869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4442 2022-12-01T10:07:27.8394319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8394677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8395092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8395433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8395764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8396219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8396550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8396975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8397313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8397631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8398125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8398774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8399162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8399499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8399756Z dist init r=0, world=2 2022-12-01T10:07:27.8399942Z dist init r=1, world=2 2022-12-01T10:07:27.8400139Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8400581Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4469 2022-12-01T10:07:27.8401048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4470 2022-12-01T10:07:27.8401500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8401817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8402245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8402590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8402913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8403377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8403703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8404131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8404461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8404798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8405289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8405855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8406235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8406587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8406851Z dist init r=1, world=2 2022-12-01T10:07:27.8407020Z dist init r=0, world=2 2022-12-01T10:07:27.8407265Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8407703Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4497 2022-12-01T10:07:27.8408168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4498 2022-12-01T10:07:27.8408609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8408936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8409364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8409694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8410031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8410509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8410839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8411251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8411593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8411927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8412418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8412925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8413307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8413654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8413900Z dist init r=1, world=2 2022-12-01T10:07:27.8414080Z dist init r=0, world=2 2022-12-01T10:07:27.8414287Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8414722Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4525 2022-12-01T10:07:27.8415171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4526 2022-12-01T10:07:27.8415616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8415947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8416362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8416709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8417041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8417505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8417817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8418353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8418702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8419036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8419513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8420076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8420466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8420797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8421058Z dist init r=1, world=2 2022-12-01T10:07:27.8421238Z dist init r=0, world=2 2022-12-01T10:07:27.8421445Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8421866Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4553 2022-12-01T10:07:27.8422333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4554 2022-12-01T10:07:27.8422787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8423106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8423535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8423880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8424215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8424673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8425001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8425427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8425770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8426092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8426584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8427104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8427474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8427820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8428073Z dist init r=1, world=2 2022-12-01T10:07:27.8428253Z dist init r=0, world=2 2022-12-01T10:07:27.8428448Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8428880Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4581 2022-12-01T10:07:27.8429340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4582 2022-12-01T10:07:27.8429775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8430102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8430556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8430900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8431219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8431684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8432013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8432466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8432792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8433123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8433617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8434124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8434506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8434852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8435111Z dist init r=0, world=2 2022-12-01T10:07:27.8435282Z dist init r=1, world=2 2022-12-01T10:07:27.8435490Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8435920Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4609 2022-12-01T10:07:27.8436368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4610 2022-12-01T10:07:27.8436823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8437150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8437577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8437908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8438436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8438916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8439244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8439660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8440006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8440346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8440824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8441348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8441736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8442089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8442337Z dist init r=1, world=2 2022-12-01T10:07:27.8442514Z dist init r=0, world=2 2022-12-01T10:07:27.8442721Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8443063Z test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4637 2022-12-01T10:07:27.8443501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4638 2022-12-01T10:07:27.8443957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8444287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8444700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8445087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8445426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8445895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8446207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8446634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8446977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8447301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8447791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8448315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8448701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8449040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8449297Z dist init r=1, world=2 2022-12-01T10:07:27.8449477Z dist init r=0, world=2 2022-12-01T10:07:27.8449671Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8450088Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4665 2022-12-01T10:07:27.8450535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4666 2022-12-01T10:07:27.8450982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8451301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8451730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8452075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8452395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8452860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8453187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8453614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8453945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8454280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8454771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8455290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8455662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8456041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8456300Z dist init r=0, world=2 2022-12-01T10:07:27.8456468Z dist init r=1, world=2 2022-12-01T10:07:27.8456675Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8457090Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4693 2022-12-01T10:07:27.8457563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4694 2022-12-01T10:07:27.8458088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8458419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8458850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8459186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8459522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8459993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8460323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8460738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8461088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8461424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8461913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8462418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8462801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8463144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8463389Z dist init r=0, world=2 2022-12-01T10:07:27.8463568Z dist init r=1, world=2 2022-12-01T10:07:27.8463775Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8464194Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4721 2022-12-01T10:07:27.8464625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4722 2022-12-01T10:07:27.8465073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8465402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8465817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8466161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8466496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8466964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8467283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8467709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8468053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8468424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8468906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8469428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8469811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8470206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8470467Z dist init r=0, world=2 2022-12-01T10:07:27.8470654Z dist init r=1, world=2 2022-12-01T10:07:27.8470863Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8471266Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4749 2022-12-01T10:07:27.8471717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4750 2022-12-01T10:07:27.8472173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8472490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8472920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8473266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8473597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8474049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8474375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8474800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8475140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8475458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8475948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8476463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8476833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8477176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8477426Z dist init r=1, world=2 2022-12-01T10:07:27.8477607Z dist init r=0, world=2 2022-12-01T10:07:27.8477798Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8478330Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4777 2022-12-01T10:07:27.8478782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4778 2022-12-01T10:07:27.8479225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8479561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8479991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8480337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8480663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8481199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8481531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8481961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8482291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8482630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8483167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8483673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8484059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8484408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8484665Z dist init r=0, world=2 2022-12-01T10:07:27.8484835Z dist init r=1, world=2 2022-12-01T10:07:27.8485042Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8485456Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4805 2022-12-01T10:07:27.8485889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4806 2022-12-01T10:07:27.8486335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8486661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8487091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8487422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8487757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8488222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8488547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8488964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8489309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8489642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8490120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8490639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8491026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8491372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8491616Z dist init r=1, world=2 2022-12-01T10:07:27.8491795Z dist init r=0, world=2 2022-12-01T10:07:27.8492002Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8492407Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4833 2022-12-01T10:07:27.8492850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4834 2022-12-01T10:07:27.8493297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8493655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8494073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8494417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8494749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8495222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8495568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8495990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8496333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8496653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8497147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8497668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8498113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8498448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8498710Z dist init r=0, world=2 2022-12-01T10:07:27.8498894Z dist init r=1, world=2 2022-12-01T10:07:27.8499092Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:07:27.8499508Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4861 2022-12-01T10:07:27.8499962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4862 2022-12-01T10:07:27.8500425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8500744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8501172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8501514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8501849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8502303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8502629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8503052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8503386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8503720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8504213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8504730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8505104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8505449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8505705Z dist init r=1, world=2 2022-12-01T10:07:27.8505873Z dist init r=0, world=2 2022-12-01T10:07:27.8506079Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8506496Z test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4889 2022-12-01T10:07:27.8506917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4890 2022-12-01T10:07:27.8507354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8507681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8508158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8508490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8508827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8509295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8509624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8510039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8510383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8510720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8511218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8511727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8512111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8512457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8512704Z dist init r=1, world=2 2022-12-01T10:07:27.8512886Z dist init r=0, world=2 2022-12-01T10:07:27.8513092Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8513471Z test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4917 2022-12-01T10:07:27.8513875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4918 2022-12-01T10:07:27.8514331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8514659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8515075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8515421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8515753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8516223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8516535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8516963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8517307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8517643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8518122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8518783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8519225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8519561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8519826Z dist init r=0, world=2 2022-12-01T10:07:27.8520010Z dist init r=1, world=2 2022-12-01T10:07:27.8520223Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8520682Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4945 2022-12-01T10:07:27.8521218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4946 2022-12-01T10:07:27.8521672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8521990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8522421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8522771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8523114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8523598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8524095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8524420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8524848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8525181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8525515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8526005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8526376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8526722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8526978Z dist init r=1, world=2 2022-12-01T10:07:27.8527158Z dist init r=0, world=2 2022-12-01T10:07:27.8527352Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8527820Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4973 2022-12-01T10:07:27.8528314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4974 2022-12-01T10:07:27.8528765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8529081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8529509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8529857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8530182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8530654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8530980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8531404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8531774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8532111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8532603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8533110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8533523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8533869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8534128Z dist init r=1, world=2 2022-12-01T10:07:27.8534296Z dist init r=0, world=2 2022-12-01T10:07:27.8534499Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8534968Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5001 2022-12-01T10:07:27.8535466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5002 2022-12-01T10:07:27.8535902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8536232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8536659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8536992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8537329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8537798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8538193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8538611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8538960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8539299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8539781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8540305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8540693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8541046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8541294Z dist init r=1, world=2 2022-12-01T10:07:27.8541474Z dist init r=0, world=2 2022-12-01T10:07:27.8541680Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8542131Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5029 2022-12-01T10:07:27.8542625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5030 2022-12-01T10:07:27.8543076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8543406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8543822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8544204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8544541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8545126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8545443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8545903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8546267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8546682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8547168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8547687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8548072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8548405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8548664Z dist init r=0, world=2 2022-12-01T10:07:27.8548846Z dist init r=1, world=2 2022-12-01T10:07:27.8549055Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8549510Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5057 2022-12-01T10:07:27.8549999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5058 2022-12-01T10:07:27.8550456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8550773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8551201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8551547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8551883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8552338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8552664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8553088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8553430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8553750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8554245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8554761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8555134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8555484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8555735Z dist init r=0, world=2 2022-12-01T10:07:27.8555916Z dist init r=1, world=2 2022-12-01T10:07:27.8556111Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8556613Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5085 2022-12-01T10:07:27.8557105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5086 2022-12-01T10:07:27.8557558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8557875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8558453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8558853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8559178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8559656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8559990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8560424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8560757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8561095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8561589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8562102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8562490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8562839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8563097Z dist init r=1, world=2 2022-12-01T10:07:27.8563270Z dist init r=0, world=2 2022-12-01T10:07:27.8563476Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8563942Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5113 2022-12-01T10:07:27.8564434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5114 2022-12-01T10:07:27.8564877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8565205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8565633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8565970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8566305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8566774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8567100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8567513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8567858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8568191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8568685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8569196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8569617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8569965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8570211Z dist init r=1, world=2 2022-12-01T10:07:27.8570391Z dist init r=0, world=2 2022-12-01T10:07:27.8570600Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8571048Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5141 2022-12-01T10:07:27.8571563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5142 2022-12-01T10:07:27.8572024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8572356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8572773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8573122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8573457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8573926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8574243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8574672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8575013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8575345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8575826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8576344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8576729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8577061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8577324Z dist init r=0, world=2 2022-12-01T10:07:27.8577503Z dist init r=1, world=2 2022-12-01T10:07:27.8577710Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8578225Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5169 2022-12-01T10:07:27.8578728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5170 2022-12-01T10:07:27.8579188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8579507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8579939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8580287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8580624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8581083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8581415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8581874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8582221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8582539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8583033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8583548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8583948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8584298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8584555Z dist init r=0, world=2 2022-12-01T10:07:27.8584741Z dist init r=1, world=2 2022-12-01T10:07:27.8584937Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8585403Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5197 2022-12-01T10:07:27.8585892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5198 2022-12-01T10:07:27.8586342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8586663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8587091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8587436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8587757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8588230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8588559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8588985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8589317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8589650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8590147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8590673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8591046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8591395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8591652Z dist init r=1, world=2 2022-12-01T10:07:27.8591821Z dist init r=0, world=2 2022-12-01T10:07:27.8592026Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8592488Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5225 2022-12-01T10:07:27.8592981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5226 2022-12-01T10:07:27.8593419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8593747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8594175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8594536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8594871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8595343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8595665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8596123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8596464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8596800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8597294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8597802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8598299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8598651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8598898Z dist init r=1, world=2 2022-12-01T10:07:27.8599083Z dist init r=0, world=2 2022-12-01T10:07:27.8599295Z skip: Need at least 2 CUDA devices (1.209s) 2022-12-01T10:07:27.8599760Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5253 2022-12-01T10:07:27.8600236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5254 2022-12-01T10:07:27.8600694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8601023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8601443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8601788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8602122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8602592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8602713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8602983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8603115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8603293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8603592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8603890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8604053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8604217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8604292Z dist init r=0, world=2 2022-12-01T10:07:27.8604352Z dist init r=1, world=2 2022-12-01T10:07:27.8604455Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8604831Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5281 2022-12-01T10:07:27.8604988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5282 2022-12-01T10:07:27.8605264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8605386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8605665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8605840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8606016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8606275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8606402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8606684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8606820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8606998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8607299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8607600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8607765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8607929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8607991Z dist init r=0, world=2 2022-12-01T10:07:27.8608068Z dist init r=1, world=2 2022-12-01T10:07:27.8608173Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8608503Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5309 2022-12-01T10:07:27.8608659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5310 2022-12-01T10:07:27.8608940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8609064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8609344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8609465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8609644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8609914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8610037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8610314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8610451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8610626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8610926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8611221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8611399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8611563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8611638Z dist init r=1, world=2 2022-12-01T10:07:27.8611710Z dist init r=0, world=2 2022-12-01T10:07:27.8611811Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8612142Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5337 2022-12-01T10:07:27.8612322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5338 2022-12-01T10:07:27.8612598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8612707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8612991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8613126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8613302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8613572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8613695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8613976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8614109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8614286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8614572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8614867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8615030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8615192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8615269Z dist init r=1, world=2 2022-12-01T10:07:27.8615340Z dist init r=0, world=2 2022-12-01T10:07:27.8615442Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8615771Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5365 2022-12-01T10:07:27.8615927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5366 2022-12-01T10:07:27.8616187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8616310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8616587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8616720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8616896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8617165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8617286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8617598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8617721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8617952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8618257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8618554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8618754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8618919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8618997Z dist init r=1, world=2 2022-12-01T10:07:27.8619074Z dist init r=0, world=2 2022-12-01T10:07:27.8619163Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8619439Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5393 2022-12-01T10:07:27.8619598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5394 2022-12-01T10:07:27.8619874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8619997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8620286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8620420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8620595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8620869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8620978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8621259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8621396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8621575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8621876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8622172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8622336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8622498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8622577Z dist init r=1, world=2 2022-12-01T10:07:27.8622637Z dist init r=0, world=2 2022-12-01T10:07:27.8622740Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8623012Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5421 2022-12-01T10:07:27.8623164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5422 2022-12-01T10:07:27.8623440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8623562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8623841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8623975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8624168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8624441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8624563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8624843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8625005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8625181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8625480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8625772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8625936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8626085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8626159Z dist init r=0, world=2 2022-12-01T10:07:27.8626232Z dist init r=1, world=2 2022-12-01T10:07:27.8626335Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8626603Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5449 2022-12-01T10:07:27.8626758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5450 2022-12-01T10:07:27.8627031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8627154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8627425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8627560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8627734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8628005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8628132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8628412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8628545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8628717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8629018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8629302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8629467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8629630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8629708Z dist init r=0, world=2 2022-12-01T10:07:27.8629783Z dist init r=1, world=2 2022-12-01T10:07:27.8629886Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8630154Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5477 2022-12-01T10:07:27.8630315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5478 2022-12-01T10:07:27.8630602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8630724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8631001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8631136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8631337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8631611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8631733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8632012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8632147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8632308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8632602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8632894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8633059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8633222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8633297Z dist init r=1, world=2 2022-12-01T10:07:27.8633370Z dist init r=0, world=2 2022-12-01T10:07:27.8633474Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8633736Z test_summon_full_params_respects_reshard_after_forward_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5505 2022-12-01T10:07:27.8633891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5506 2022-12-01T10:07:27.8634162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8634282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8634561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8634697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8634878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8635150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8635273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8635539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8635673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8635849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8636144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8636440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8636605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8636767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8636841Z dist init r=1, world=2 2022-12-01T10:07:27.8636928Z dist init r=0, world=2 2022-12-01T10:07:27.8637032Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8637307Z test_summon_full_params_respects_reshard_after_forward_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5533 2022-12-01T10:07:27.8637460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5534 2022-12-01T10:07:27.8637733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8637885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8638162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8638418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8638598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8638860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8638984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8639264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8639397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8639576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8639878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8640176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8640343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8640510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8640572Z dist init r=0, world=2 2022-12-01T10:07:27.8640645Z dist init r=1, world=2 2022-12-01T10:07:27.8640752Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8640975Z test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5561 2022-12-01T10:07:27.8641130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5562 2022-12-01T10:07:27.8641402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8641526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8641808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8641933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8642106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8642376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8642498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8642780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8642915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8643091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:07:27.8643389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8643774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:07:27.8643939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:07:27.8644103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8644180Z dist init r=1, world=2 2022-12-01T10:07:27.8644255Z dist init r=0, world=2 2022-12-01T10:07:27.8644409Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:07:27.8644715Z test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5589 2022-12-01T10:07:27.8644990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8645115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8645387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8645525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8645701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8646001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:07:27.8646165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8646241Z dist init r=0, world=1 2022-12-01T10:07:27.8646347Z skip: Need at least 2 CUDA devices (1.207s) 2022-12-01T10:07:27.8646649Z test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5603 2022-12-01T10:07:27.8646927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8647038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8647320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8647456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8647631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8647929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:07:27.8648090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8648169Z dist init r=0, world=1 2022-12-01T10:07:27.8648270Z skip: Need at least 2 CUDA devices (1.107s) 2022-12-01T10:07:27.8648560Z test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5617 2022-12-01T10:07:27.8648831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8648952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8649236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8649369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8649545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8649843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:07:27.8650031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8650106Z dist init r=0, world=1 2022-12-01T10:07:27.8650197Z skip: Need at least 2 CUDA devices (1.107s) 2022-12-01T10:07:27.8650496Z test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5631 2022-12-01T10:07:27.8650769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8650920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8651200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8651332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8651509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8651805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:07:27.8651966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8652029Z dist init r=0, world=1 2022-12-01T10:07:27.8652129Z skip: Need at least 2 CUDA devices (1.107s) 2022-12-01T10:07:27.8652428Z test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5645 2022-12-01T10:07:27.8652700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8652822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8653105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8653240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8653415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8653699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:07:27.8653863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8653942Z dist init r=0, world=1 2022-12-01T10:07:27.8654045Z skip: Need at least 2 CUDA devices (1.107s) 2022-12-01T10:07:27.8654345Z test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5659 2022-12-01T10:07:27.8654621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8654744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8655023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8655157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8655320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8655618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:07:27.8655782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8655858Z dist init r=0, world=1 2022-12-01T10:07:27.8655960Z skip: Need at least 2 CUDA devices (1.107s) 2022-12-01T10:07:27.8656287Z test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5673 2022-12-01T10:07:27.8656563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8656685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8656963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8657112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8657290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8657586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:07:27.8657749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8657828Z dist init r=0, world=1 2022-12-01T10:07:27.8657989Z skip: Need at least 2 CUDA devices (1.107s) 2022-12-01T10:07:27.8658289Z test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5687 2022-12-01T10:07:27.8658566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:07:27.8658682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:07:27.8658965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:07:27.8659101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:07:27.8659277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:07:27.8659581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:07:27.8659748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:07:27.8659824Z dist init r=0, world=1 2022-12-01T10:07:27.8659928Z skip: Need at least 2 CUDA devices (1.107s) 2022-12-01T10:07:27.8659937Z 2022-12-01T10:07:27.8660143Z ---------------------------------------------------------------------- 2022-12-01T10:07:27.8660212Z Ran 73 tests in 87.848s 2022-12-01T10:07:27.8660217Z 2022-12-01T10:07:27.8660290Z OK (skipped=73) 2022-12-01T10:07:27.8660295Z 2022-12-01T10:07:27.8660379Z Generating XML reports... 2022-12-01T10:07:27.8660743Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221201100559.xml 2022-12-01T10:07:27.8661129Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221201100559.xml 2022-12-01T10:07:27.8661137Z 2022-12-01T10:07:27.8661488Z ##[endgroup] 2022-12-01T10:07:27.8661881Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_4sdjf84n) 2022-12-01T10:07:27.8661887Z 2022-12-01T10:07:27.8662074Z Running distributed/test_c10d_nccl ... [2022-12-01 10:07:27.814536] 2022-12-01T10:07:27.8662435Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:07:27.814951] 2022-12-01T10:10:59.3561568Z 2022-12-01T10:10:59.3562538Z Expand the folded group to see the log file of distributed/test_c10d_nccl 2022-12-01T10:10:59.3563586Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_290344ik) 2022-12-01T10:10:59.3648009Z , <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]> 2022-12-01T10:10:59.3649626Z test_all_reduce_coalesced_nccl (__main__.CommTest) 2022-12-01T10:10:59.3649877Z test_broadcast_coalesced_nccl (__main__.CommTest) 2022-12-01T10:10:59.3650095Z test_nccl_barrier (__main__.CommTest) 2022-12-01T10:10:59.3650321Z test_nccl_barrier_device_ids (__main__.CommTest) 2022-12-01T10:10:59.3650586Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) 2022-12-01T10:10:59.3650831Z test_nccl_barrier_timeout (__main__.CommTest) 2022-12-01T10:10:59.3651078Z test_nccl_barrier_timeout_new_group (__main__.CommTest) 2022-12-01T10:10:59.3651345Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) 2022-12-01T10:10:59.3654401Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) 2022-12-01T10:10:59.3654845Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) 2022-12-01T10:10:59.3655305Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) 2022-12-01T10:10:59.3678975Z test_nncl_rank_membership (__main__.CommTest) 2022-12-01T10:10:59.3679420Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) 2022-12-01T10:10:59.3679887Z test_sequence_num_incremented_nccl_default (__main__.CommTest) 2022-12-01T10:10:59.3680388Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) 2022-12-01T10:10:59.3680899Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) 2022-12-01T10:10:59.3681356Z test_sequence_num_set_nccl_new_group (__main__.CommTest) 2022-12-01T10:10:59.3681804Z test_tensor_dtype_complex (__main__.CommTest) 2022-12-01T10:10:59.3682222Z test_tensor_dtype_mismatch (__main__.CommTest) 2022-12-01T10:10:59.3683364Z , <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]> 2022-12-01T10:10:59.3684575Z test_allgather_work_wait_gpu (__main__.CompilerTest) 2022-12-01T10:10:59.3685030Z test_allreduce_work_wait_gpu (__main__.CompilerTest) 2022-12-01T10:10:59.3685500Z test_broadcast_work_wait_gpu (__main__.CompilerTest) 2022-12-01T10:10:59.3685977Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) 2022-12-01T10:10:59.3686423Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) 2022-12-01T10:10:59.3686924Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) 2022-12-01T10:10:59.3687332Z test_scatter_work_wait_gpu (__main__.CompilerTest) 2022-12-01T10:10:59.3698206Z , <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-12-01T10:10:59.3706886Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3707248Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3707600Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3707947Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3708272Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3708710Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3709016Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3709352Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3709680Z test_channels_last_contig (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3709991Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3710344Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3710707Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3711067Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3711430Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3711822Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3712197Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3712544Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3712897Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3713260Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3713638Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3714050Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3714597Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3714955Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3715292Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3715652Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3716045Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3716397Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3716717Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3717036Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3717352Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3717659Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3717987Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3718471Z test_failure_recovery (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3718809Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3719151Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3719504Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3719869Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3720235Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3720616Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3721329Z test_fp16 (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3721832Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3722375Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3722950Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3723569Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3724197Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3724724Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3725299Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3725938Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3726568Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3727151Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3727709Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3728204Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3728754Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3729341Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3729939Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3730513Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3731085Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3731953Z test_no_grad (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3732578Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3733072Z test_pass_default_pg (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3733582Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3734139Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3734658Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3735274Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3735735Z 2022-12-01T10:10:59.3737400Z , <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]> 2022-12-01T10:10:59.3738992Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3739490Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3739997Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3740480Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3741051Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3741593Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3742144Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3742659Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3743152Z test_nccl_timeout (__main__.NcclErrorHandlingTest) 2022-12-01T10:10:59.3743612Z ]> 2022-12-01T10:10:59.3744063Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-12-01T10:10:59.3744465Z ]> 2022-12-01T10:10:59.3744792Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) 2022-12-01T10:10:59.3746284Z , <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]> 2022-12-01T10:10:59.3747813Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3748163Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3748437Z test_allgather_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3748687Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3748946Z test_barrier (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3749204Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3749455Z test_empty_tensors (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3749751Z test_gather_checks (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3750011Z test_gather_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3750268Z test_gather_stress (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3750507Z test_reduce_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3750781Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3751071Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3751336Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3751603Z test_scatter_checks (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3751862Z test_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3752110Z test_scatter_stress (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3752367Z test_send_recv (__main__.ProcessGroupNCCLTest) 2022-12-01T10:10:59.3752680Z ]> 2022-12-01T10:10:59.3752974Z test_common_errors (__main__.RendezvousEnvTest) 2022-12-01T10:10:59.3753210Z 2022-12-01T10:10:59.3753525Z ]> 2022-12-01T10:10:59.3753841Z test_default_store_timeout_nccl (__main__.TimeoutTest) 2022-12-01T10:10:59.3754305Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3754783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3755120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3755538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3755884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3756050Z 2022-12-01T10:10:59.3756125Z Running tests... 2022-12-01T10:10:59.3756434Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3756770Z test_all_reduce_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.3756972Z 2022-12-01T10:10:59.3757166Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3757405Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3757519Z 2022-12-01T10:10:59.3757590Z OK (skipped=1) 2022-12-01T10:10:59.3757684Z 2022-12-01T10:10:59.3757772Z Generating XML reports... 2022-12-01T10:10:59.3758180Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100730.xml 2022-12-01T10:10:59.3758794Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3759250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3759585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3760022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3760373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3760524Z 2022-12-01T10:10:59.3760597Z Running tests... 2022-12-01T10:10:59.3760900Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3761322Z test_broadcast_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.3761526Z 2022-12-01T10:10:59.3761707Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3761945Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3762058Z 2022-12-01T10:10:59.3762130Z OK (skipped=1) 2022-12-01T10:10:59.3762237Z 2022-12-01T10:10:59.3762322Z Generating XML reports... 2022-12-01T10:10:59.3762716Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100731.xml 2022-12-01T10:10:59.3763205Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3763667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3763985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3764421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3764767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3764930Z 2022-12-01T10:10:59.3765002Z Running tests... 2022-12-01T10:10:59.3765298Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3765628Z test_nccl_barrier (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-12-01T10:10:59.3765815Z 2022-12-01T10:10:59.3766013Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3766237Z Ran 1 test in 0.002s 2022-12-01T10:10:59.3766350Z 2022-12-01T10:10:59.3766421Z OK (skipped=1) 2022-12-01T10:10:59.3766528Z 2022-12-01T10:10:59.3766613Z Generating XML reports... 2022-12-01T10:10:59.3767018Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100732.xml 2022-12-01T10:10:59.3767452Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3767909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3768237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3768655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3769003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3769168Z 2022-12-01T10:10:59.3769241Z Running tests... 2022-12-01T10:10:59.3769543Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3769871Z test_nccl_barrier_device_ids (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3770069Z 2022-12-01T10:10:59.3770264Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3770500Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3770613Z 2022-12-01T10:10:59.3770687Z OK (skipped=1) 2022-12-01T10:10:59.3770781Z 2022-12-01T10:10:59.3770865Z Generating XML reports... 2022-12-01T10:10:59.3771268Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100734.xml 2022-12-01T10:10:59.3771715Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3772166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3772499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3772926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3773275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3773426Z 2022-12-01T10:10:59.3773497Z Running tests... 2022-12-01T10:10:59.3773799Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3774192Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3774410Z 2022-12-01T10:10:59.3775518Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3775739Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3775851Z 2022-12-01T10:10:59.3775922Z OK (skipped=1) 2022-12-01T10:10:59.3776033Z 2022-12-01T10:10:59.3776156Z Generating XML reports... 2022-12-01T10:10:59.3776544Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100735.xml 2022-12-01T10:10:59.3777074Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3777535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3777867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3778285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3778632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3778800Z 2022-12-01T10:10:59.3778874Z Running tests... 2022-12-01T10:10:59.3779160Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3779504Z test_nccl_barrier_timeout (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.3779705Z 2022-12-01T10:10:59.3779899Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3780143Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3780241Z 2022-12-01T10:10:59.3780313Z OK (skipped=1) 2022-12-01T10:10:59.3780422Z 2022-12-01T10:10:59.3780506Z Generating XML reports... 2022-12-01T10:10:59.3780911Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100737.xml 2022-12-01T10:10:59.3781349Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3781809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3782142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3782573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3782909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3783075Z 2022-12-01T10:10:59.3783147Z Running tests... 2022-12-01T10:10:59.3783451Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3783786Z test_nccl_barrier_timeout_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.3783994Z 2022-12-01T10:10:59.3784191Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3784428Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3784542Z 2022-12-01T10:10:59.3784612Z OK (skipped=1) 2022-12-01T10:10:59.3784706Z 2022-12-01T10:10:59.3784790Z Generating XML reports... 2022-12-01T10:10:59.3785196Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100738.xml 2022-12-01T10:10:59.3785644Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3786097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3786430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3786858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3787204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3787368Z 2022-12-01T10:10:59.3787462Z Running tests... 2022-12-01T10:10:59.3787764Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3788126Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.3788344Z 2022-12-01T10:10:59.3788536Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3788762Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3788912Z 2022-12-01T10:10:59.3788986Z OK (skipped=1) 2022-12-01T10:10:59.3789095Z 2022-12-01T10:10:59.3789180Z Generating XML reports... 2022-12-01T10:10:59.3789572Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100739.xml 2022-12-01T10:10:59.3790024Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3790489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3790819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3791234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3791580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3791744Z 2022-12-01T10:10:59.3791816Z Running tests... 2022-12-01T10:10:59.3792106Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3792468Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3792674Z 2022-12-01T10:10:59.3792870Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3793114Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3793212Z 2022-12-01T10:10:59.3793284Z OK (skipped=1) 2022-12-01T10:10:59.3793392Z 2022-12-01T10:10:59.3793478Z Generating XML reports... 2022-12-01T10:10:59.3793882Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100741.xml 2022-12-01T10:10:59.3794318Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3794777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3795108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3795539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3795872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3796035Z 2022-12-01T10:10:59.3796110Z Running tests... 2022-12-01T10:10:59.3796410Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3796753Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3796961Z 2022-12-01T10:10:59.3797157Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3797396Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3797507Z 2022-12-01T10:10:59.3797580Z OK (skipped=1) 2022-12-01T10:10:59.3797689Z 2022-12-01T10:10:59.3797760Z Generating XML reports... 2022-12-01T10:10:59.3798161Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100742.xml 2022-12-01T10:10:59.3798849Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3799314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3799629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3800059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3800470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3800639Z 2022-12-01T10:10:59.3800699Z Running tests... 2022-12-01T10:10:59.3800999Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3801351Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3801558Z 2022-12-01T10:10:59.3801795Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3802018Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3802131Z 2022-12-01T10:10:59.3802204Z OK (skipped=1) 2022-12-01T10:10:59.3802311Z 2022-12-01T10:10:59.3802398Z Generating XML reports... 2022-12-01T10:10:59.3802792Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100743.xml 2022-12-01T10:10:59.3803244Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3803706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3804037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3804454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3804804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3804971Z 2022-12-01T10:10:59.3805047Z Running tests... 2022-12-01T10:10:59.3805336Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3805679Z test_nncl_rank_membership (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3805873Z 2022-12-01T10:10:59.3806070Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3806306Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3806417Z 2022-12-01T10:10:59.3806480Z OK (skipped=1) 2022-12-01T10:10:59.3806585Z 2022-12-01T10:10:59.3806669Z Generating XML reports... 2022-12-01T10:10:59.3807073Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100745.xml 2022-12-01T10:10:59.3807512Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3807972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3808305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3808733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3809142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3809307Z 2022-12-01T10:10:59.3809381Z Running tests... 2022-12-01T10:10:59.3809687Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3810099Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.3810301Z 2022-12-01T10:10:59.3810500Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3810734Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3810847Z 2022-12-01T10:10:59.3810919Z OK (skipped=1) 2022-12-01T10:10:59.3811025Z 2022-12-01T10:10:59.3811098Z Generating XML reports... 2022-12-01T10:10:59.3811503Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100746.xml 2022-12-01T10:10:59.3811953Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3812414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3812731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3813196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3813547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3813711Z 2022-12-01T10:10:59.3813769Z Running tests... 2022-12-01T10:10:59.3814075Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3814436Z test_sequence_num_incremented_nccl_default (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3814682Z 2022-12-01T10:10:59.3814896Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3815156Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3815266Z 2022-12-01T10:10:59.3815341Z OK (skipped=1) 2022-12-01T10:10:59.3815448Z 2022-12-01T10:10:59.3815532Z Generating XML reports... 2022-12-01T10:10:59.3815929Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100748.xml 2022-12-01T10:10:59.3816383Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3816896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3817232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3817649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3818001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3818165Z 2022-12-01T10:10:59.3818238Z Running tests... 2022-12-01T10:10:59.3818526Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3818889Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3819103Z 2022-12-01T10:10:59.3819300Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3819539Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3819653Z 2022-12-01T10:10:59.3819710Z OK (skipped=1) 2022-12-01T10:10:59.3819817Z 2022-12-01T10:10:59.3819905Z Generating XML reports... 2022-12-01T10:10:59.3820311Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100749.xml 2022-12-01T10:10:59.3820760Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3821206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3821537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3821965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3822299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3822467Z 2022-12-01T10:10:59.3822542Z Running tests... 2022-12-01T10:10:59.3822843Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3823197Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3823389Z 2022-12-01T10:10:59.3823586Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3823828Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3823939Z 2022-12-01T10:10:59.3824012Z OK (skipped=1) 2022-12-01T10:10:59.3824120Z 2022-12-01T10:10:59.3824191Z Generating XML reports... 2022-12-01T10:10:59.3824593Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100750.xml 2022-12-01T10:10:59.3825040Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3825557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3825872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3826301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3826649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3826815Z 2022-12-01T10:10:59.3826889Z Running tests... 2022-12-01T10:10:59.3827212Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3827564Z test_sequence_num_set_nccl_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3827770Z 2022-12-01T10:10:59.3827966Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3828191Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3828305Z 2022-12-01T10:10:59.3828378Z OK (skipped=1) 2022-12-01T10:10:59.3828485Z 2022-12-01T10:10:59.3828571Z Generating XML reports... 2022-12-01T10:10:59.3828961Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100752.xml 2022-12-01T10:10:59.3829410Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3829867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3830198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3830619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3830964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3831127Z 2022-12-01T10:10:59.3831200Z Running tests... 2022-12-01T10:10:59.3831501Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3831830Z test_tensor_dtype_complex (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3832025Z 2022-12-01T10:10:59.3832220Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3832457Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3832571Z 2022-12-01T10:10:59.3832631Z OK (skipped=1) 2022-12-01T10:10:59.3832738Z 2022-12-01T10:10:59.3832826Z Generating XML reports... 2022-12-01T10:10:59.3833227Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100753.xml 2022-12-01T10:10:59.3833683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3834129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3834458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3834892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3835226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3835389Z 2022-12-01T10:10:59.3835460Z Running tests... 2022-12-01T10:10:59.3835761Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3836103Z test_tensor_dtype_mismatch (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3836300Z 2022-12-01T10:10:59.3836484Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3836719Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3836831Z 2022-12-01T10:10:59.3836902Z OK (skipped=1) 2022-12-01T10:10:59.3837010Z 2022-12-01T10:10:59.3837080Z Generating XML reports... 2022-12-01T10:10:59.3837485Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100755.xml 2022-12-01T10:10:59.3837935Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3838624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3838949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3839384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3839732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3839937Z 2022-12-01T10:10:59.3840010Z Running tests... 2022-12-01T10:10:59.3840298Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3840681Z test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5912 2022-12-01T10:10:59.3841064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5913 2022-12-01T10:10:59.3841503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3841834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3842263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3842608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3842918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3843377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3843703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3844115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3844460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3844787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3845077Z skip: Need at least 2 CUDA devices (1.741s) 2022-12-01T10:10:59.3845199Z 2022-12-01T10:10:59.3845399Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3845637Z Ran 1 test in 1.741s 2022-12-01T10:10:59.3845749Z 2022-12-01T10:10:59.3845820Z OK (skipped=1) 2022-12-01T10:10:59.3845928Z 2022-12-01T10:10:59.3846002Z Generating XML reports... 2022-12-01T10:10:59.3846418Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100756.xml 2022-12-01T10:10:59.3846882Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3847337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3847656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3848088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3848434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3848599Z 2022-12-01T10:10:59.3848672Z Running tests... 2022-12-01T10:10:59.3848960Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3849344Z test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5945 2022-12-01T10:10:59.3849732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5946 2022-12-01T10:10:59.3850169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3850498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3850962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3851310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3851620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3852081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3852409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3852856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3853206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3853530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3853821Z skip: Need at least 2 CUDA devices (1.742s) 2022-12-01T10:10:59.3853943Z 2022-12-01T10:10:59.3854145Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3854389Z Ran 1 test in 1.742s 2022-12-01T10:10:59.3854503Z 2022-12-01T10:10:59.3854575Z OK (skipped=1) 2022-12-01T10:10:59.3854682Z 2022-12-01T10:10:59.3854754Z Generating XML reports... 2022-12-01T10:10:59.3855173Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100759.xml 2022-12-01T10:10:59.3855632Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3856096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3856411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3856896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3857246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3857413Z 2022-12-01T10:10:59.3857486Z Running tests... 2022-12-01T10:10:59.3857776Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3858156Z test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5978 2022-12-01T10:10:59.3858545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5979 2022-12-01T10:10:59.3858984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3859316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3859742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3860087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3860401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3860861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3861189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3861603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3861946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3862272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3862562Z skip: Need at least 2 CUDA devices (1.741s) 2022-12-01T10:10:59.3862682Z 2022-12-01T10:10:59.3862883Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3863119Z Ran 1 test in 1.742s 2022-12-01T10:10:59.3863231Z 2022-12-01T10:10:59.3863304Z OK (skipped=1) 2022-12-01T10:10:59.3863410Z 2022-12-01T10:10:59.3863517Z Generating XML reports... 2022-12-01T10:10:59.3863938Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100802.xml 2022-12-01T10:10:59.3864398Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3864859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3865183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3865646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3865992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3866156Z 2022-12-01T10:10:59.3866215Z Running tests... 2022-12-01T10:10:59.3866517Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3866909Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6011 2022-12-01T10:10:59.3867302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6012 2022-12-01T10:10:59.3867741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3868071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3868503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3868851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3869162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3869618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3869949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3870362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3870708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3871027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3871314Z skip: Need at least 2 CUDA devices (1.746s) 2022-12-01T10:10:59.3871440Z 2022-12-01T10:10:59.3871641Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3871881Z Ran 1 test in 1.746s 2022-12-01T10:10:59.3871993Z 2022-12-01T10:10:59.3872066Z OK (skipped=1) 2022-12-01T10:10:59.3872174Z 2022-12-01T10:10:59.3872245Z Generating XML reports... 2022-12-01T10:10:59.3872662Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100806.xml 2022-12-01T10:10:59.3873127Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3873585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3873902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3874328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3874674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3874842Z 2022-12-01T10:10:59.3874901Z Running tests... 2022-12-01T10:10:59.3875204Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3875590Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6044 2022-12-01T10:10:59.3875978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6045 2022-12-01T10:10:59.3876449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3876780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3877210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3877558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3877898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3878504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3878838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3879253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3879603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3879932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3880225Z skip: Need at least 2 CUDA devices (1.740s) 2022-12-01T10:10:59.3880346Z 2022-12-01T10:10:59.3880547Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3880790Z Ran 1 test in 1.740s 2022-12-01T10:10:59.3880902Z 2022-12-01T10:10:59.3880974Z OK (skipped=1) 2022-12-01T10:10:59.3881088Z 2022-12-01T10:10:59.3881160Z Generating XML reports... 2022-12-01T10:10:59.3881581Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100809.xml 2022-12-01T10:10:59.3882042Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3882501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3882821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3883247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3883588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3883751Z 2022-12-01T10:10:59.3883810Z Running tests... 2022-12-01T10:10:59.3884113Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3884507Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6077 2022-12-01T10:10:59.3884897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6078 2022-12-01T10:10:59.3885337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3885668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3886103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3886449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3886760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3887220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3887550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3887966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3888308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3888632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3888923Z skip: Need at least 2 CUDA devices (1.744s) 2022-12-01T10:10:59.3889102Z 2022-12-01T10:10:59.3889305Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3889547Z Ran 1 test in 1.745s 2022-12-01T10:10:59.3889660Z 2022-12-01T10:10:59.3889733Z OK (skipped=1) 2022-12-01T10:10:59.3889841Z 2022-12-01T10:10:59.3889913Z Generating XML reports... 2022-12-01T10:10:59.3890331Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100812.xml 2022-12-01T10:10:59.3890847Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3891307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3891627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3892058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3892408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3892572Z 2022-12-01T10:10:59.3892632Z Running tests... 2022-12-01T10:10:59.3892934Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3893318Z test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6110 2022-12-01T10:10:59.3893700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6111 2022-12-01T10:10:59.3894140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3894469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3894898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3895230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3895556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3896015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3896343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3896755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3897167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3897489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3897783Z skip: Need at least 2 CUDA devices (1.742s) 2022-12-01T10:10:59.3897905Z 2022-12-01T10:10:59.3898102Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3898344Z Ran 1 test in 1.742s 2022-12-01T10:10:59.3898458Z 2022-12-01T10:10:59.3898530Z OK (skipped=1) 2022-12-01T10:10:59.3898640Z 2022-12-01T10:10:59.3898713Z Generating XML reports... 2022-12-01T10:10:59.3899136Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100815.xml 2022-12-01T10:10:59.3899604Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3900065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3900385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3900811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3901156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3901321Z 2022-12-01T10:10:59.3901380Z Running tests... 2022-12-01T10:10:59.3901682Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3902104Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3902346Z 2022-12-01T10:10:59.3902542Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3902768Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3902878Z 2022-12-01T10:10:59.3902952Z OK (skipped=1) 2022-12-01T10:10:59.3903060Z 2022-12-01T10:10:59.3903146Z Generating XML reports... 2022-12-01T10:10:59.3903635Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100819.xml 2022-12-01T10:10:59.3904137Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3904597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3904929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3905343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3905690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3905854Z 2022-12-01T10:10:59.3905928Z Running tests... 2022-12-01T10:10:59.3906215Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3906622Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3906875Z 2022-12-01T10:10:59.3907068Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3907309Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3907423Z 2022-12-01T10:10:59.3907480Z OK (skipped=1) 2022-12-01T10:10:59.3907588Z 2022-12-01T10:10:59.3907674Z Generating XML reports... 2022-12-01T10:10:59.3908136Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100820.xml 2022-12-01T10:10:59.3908634Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3909078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3909409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3909837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3910172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3910337Z 2022-12-01T10:10:59.3910409Z Running tests... 2022-12-01T10:10:59.3910709Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3911096Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3911332Z 2022-12-01T10:10:59.3911515Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3911753Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3911865Z 2022-12-01T10:10:59.3911938Z OK (skipped=1) 2022-12-01T10:10:59.3912044Z 2022-12-01T10:10:59.3912115Z Generating XML reports... 2022-12-01T10:10:59.3912583Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100821.xml 2022-12-01T10:10:59.3913085Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3913548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3913867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3914294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3914673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3914841Z 2022-12-01T10:10:59.3914913Z Running tests... 2022-12-01T10:10:59.3915199Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3915598Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3915848Z 2022-12-01T10:10:59.3916042Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3916296Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3916408Z 2022-12-01T10:10:59.3916481Z OK (skipped=1) 2022-12-01T10:10:59.3916587Z 2022-12-01T10:10:59.3916675Z Generating XML reports... 2022-12-01T10:10:59.3917140Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100823.xml 2022-12-01T10:10:59.3917630Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3918094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3918567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3918985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3919336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3919506Z 2022-12-01T10:10:59.3919580Z Running tests... 2022-12-01T10:10:59.3919884Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3920256Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3920493Z 2022-12-01T10:10:59.3920687Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3920926Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3921040Z 2022-12-01T10:10:59.3921113Z OK (skipped=1) 2022-12-01T10:10:59.3921207Z 2022-12-01T10:10:59.3921294Z Generating XML reports... 2022-12-01T10:10:59.3921756Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100824.xml 2022-12-01T10:10:59.3922255Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3922705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3923038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3923467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3923821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3923970Z 2022-12-01T10:10:59.3924044Z Running tests... 2022-12-01T10:10:59.3924347Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3924730Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3924963Z 2022-12-01T10:10:59.3925142Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3925385Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3925497Z 2022-12-01T10:10:59.3925569Z OK (skipped=1) 2022-12-01T10:10:59.3925678Z 2022-12-01T10:10:59.3925763Z Generating XML reports... 2022-12-01T10:10:59.3926215Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100825.xml 2022-12-01T10:10:59.3926719Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3927184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3927574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3928008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3928356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3928521Z 2022-12-01T10:10:59.3928595Z Running tests... 2022-12-01T10:10:59.3928885Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3929305Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3929539Z 2022-12-01T10:10:59.3929734Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3929975Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3930073Z 2022-12-01T10:10:59.3930145Z OK (skipped=1) 2022-12-01T10:10:59.3930252Z 2022-12-01T10:10:59.3930336Z Generating XML reports... 2022-12-01T10:10:59.3930800Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100827.xml 2022-12-01T10:10:59.3931289Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3931752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3932086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3932521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3932854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3933105Z 2022-12-01T10:10:59.3933177Z Running tests... 2022-12-01T10:10:59.3933480Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3933860Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.3934108Z 2022-12-01T10:10:59.3934300Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3934538Z Ran 1 test in 0.000s 2022-12-01T10:10:59.3934650Z 2022-12-01T10:10:59.3934723Z OK (skipped=1) 2022-12-01T10:10:59.3934830Z 2022-12-01T10:10:59.3934901Z Generating XML reports... 2022-12-01T10:10:59.3935363Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100828.xml 2022-12-01T10:10:59.3935863Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3936325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3936644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3937139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3937493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3937659Z 2022-12-01T10:10:59.3937717Z Running tests... 2022-12-01T10:10:59.3938020Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3938397Z test_channels_last_contig (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.3938628Z 2022-12-01T10:10:59.3938822Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3939050Z Ran 1 test in 0.001s 2022-12-01T10:10:59.3939164Z 2022-12-01T10:10:59.3939238Z OK (skipped=1) 2022-12-01T10:10:59.3939345Z 2022-12-01T10:10:59.3939430Z Generating XML reports... 2022-12-01T10:10:59.3939880Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100830.xml 2022-12-01T10:10:59.3940415Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3940877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3941209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3941625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3941969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3942167Z 2022-12-01T10:10:59.3942239Z Running tests... 2022-12-01T10:10:59.3942530Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3942861Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3943392Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6233 2022-12-01T10:10:59.3943801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6234 2022-12-01T10:10:59.3944241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3944572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3945003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3945352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3945662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3946122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3946451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3946867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3947212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3947536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3947828Z skip: Need at least 2 CUDA devices (1.741s) 2022-12-01T10:10:59.3947952Z 2022-12-01T10:10:59.3948148Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3948388Z Ran 1 test in 1.742s 2022-12-01T10:10:59.3948501Z 2022-12-01T10:10:59.3948575Z OK (skipped=1) 2022-12-01T10:10:59.3948683Z 2022-12-01T10:10:59.3948754Z Generating XML reports... 2022-12-01T10:10:59.3949219Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100831.xml 2022-12-01T10:10:59.3949721Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3950186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3950504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3950932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3951277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3951441Z 2022-12-01T10:10:59.3951517Z Running tests... 2022-12-01T10:10:59.3951809Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3952150Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3952583Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6266 2022-12-01T10:10:59.3952973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6267 2022-12-01T10:10:59.3953458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3953794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3954226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3954560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3954912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3955373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3955688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3956116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3956462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3956785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3957060Z skip: Need at least 2 CUDA devices (1.746s) 2022-12-01T10:10:59.3957197Z 2022-12-01T10:10:59.3957395Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3957637Z Ran 1 test in 1.746s 2022-12-01T10:10:59.3957750Z 2022-12-01T10:10:59.3957811Z OK (skipped=1) 2022-12-01T10:10:59.3957919Z 2022-12-01T10:10:59.3958004Z Generating XML reports... 2022-12-01T10:10:59.3958630Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100834.xml 2022-12-01T10:10:59.3959139Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3959585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3959920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3960346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3960685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3960848Z 2022-12-01T10:10:59.3960921Z Running tests... 2022-12-01T10:10:59.3961226Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3961573Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3961982Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6299 2022-12-01T10:10:59.3962377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6300 2022-12-01T10:10:59.3962834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3963165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3963579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3963927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3964252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3964701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3965031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3965459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3965806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3966170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3966464Z skip: Need at least 2 CUDA devices (1.748s) 2022-12-01T10:10:59.3966602Z 2022-12-01T10:10:59.3966803Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3967031Z Ran 1 test in 1.748s 2022-12-01T10:10:59.3967144Z 2022-12-01T10:10:59.3967217Z OK (skipped=1) 2022-12-01T10:10:59.3967326Z 2022-12-01T10:10:59.3967412Z Generating XML reports... 2022-12-01T10:10:59.3967918Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100838.xml 2022-12-01T10:10:59.3968402Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3968867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3969198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3969616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3969966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3970134Z 2022-12-01T10:10:59.3970209Z Running tests... 2022-12-01T10:10:59.3970512Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3970836Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3971258Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6332 2022-12-01T10:10:59.3971652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6333 2022-12-01T10:10:59.3972106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3972422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3972857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3973206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3973517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3973979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3974309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3974737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3975065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3975388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3975680Z skip: Need at least 2 CUDA devices (1.748s) 2022-12-01T10:10:59.3975816Z 2022-12-01T10:10:59.3976000Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3976239Z Ran 1 test in 1.748s 2022-12-01T10:10:59.3976352Z 2022-12-01T10:10:59.3976425Z OK (skipped=1) 2022-12-01T10:10:59.3976532Z 2022-12-01T10:10:59.3976617Z Generating XML reports... 2022-12-01T10:10:59.3977126Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100841.xml 2022-12-01T10:10:59.3977635Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3978100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3978415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3978849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3979237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3979405Z 2022-12-01T10:10:59.3979478Z Running tests... 2022-12-01T10:10:59.3979769Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3980130Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3980671Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6365 2022-12-01T10:10:59.3981102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6366 2022-12-01T10:10:59.3981543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3981871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3982303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3982639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3982963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3983425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3983753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3984171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3984516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3984841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3985117Z skip: Need at least 2 CUDA devices (1.747s) 2022-12-01T10:10:59.3985256Z 2022-12-01T10:10:59.3985458Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3985700Z Ran 1 test in 1.747s 2022-12-01T10:10:59.3985812Z 2022-12-01T10:10:59.3985884Z OK (skipped=1) 2022-12-01T10:10:59.3985979Z 2022-12-01T10:10:59.3986066Z Generating XML reports... 2022-12-01T10:10:59.3986533Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100844.xml 2022-12-01T10:10:59.3987039Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3987495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3987828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3988256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3988600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3988765Z 2022-12-01T10:10:59.3988825Z Running tests... 2022-12-01T10:10:59.3989125Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3989483Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3990027Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6398 2022-12-01T10:10:59.3990415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6399 2022-12-01T10:10:59.3990868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3991201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3991613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3991991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3992317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.3992774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3993089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3993518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3993903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3994214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.3994507Z skip: Need at least 2 CUDA devices (1.742s) 2022-12-01T10:10:59.3994643Z 2022-12-01T10:10:59.3994844Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3995087Z Ran 1 test in 1.742s 2022-12-01T10:10:59.3995186Z 2022-12-01T10:10:59.3995257Z OK (skipped=1) 2022-12-01T10:10:59.3995365Z 2022-12-01T10:10:59.3995451Z Generating XML reports... 2022-12-01T10:10:59.3995919Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100847.xml 2022-12-01T10:10:59.3996409Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.3996875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.3997208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.3997638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.3997968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.3998133Z 2022-12-01T10:10:59.3998323Z Running tests... 2022-12-01T10:10:59.3998633Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.3998977Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.3999514Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6431 2022-12-01T10:10:59.3999925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6432 2022-12-01T10:10:59.4000386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4000707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4001135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4001484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4001811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4002261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4002591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4003022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4003368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4003679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4003970Z skip: Need at least 2 CUDA devices (1.742s) 2022-12-01T10:10:59.4004106Z 2022-12-01T10:10:59.4004306Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4004534Z Ran 1 test in 1.742s 2022-12-01T10:10:59.4004646Z 2022-12-01T10:10:59.4004719Z OK (skipped=1) 2022-12-01T10:10:59.4004877Z 2022-12-01T10:10:59.4004965Z Generating XML reports... 2022-12-01T10:10:59.4005419Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100851.xml 2022-12-01T10:10:59.4005922Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4006385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4006757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4007171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4007518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4007682Z 2022-12-01T10:10:59.4007756Z Running tests... 2022-12-01T10:10:59.4008054Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4008381Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4008925Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6464 2022-12-01T10:10:59.4009340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6465 2022-12-01T10:10:59.4009777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4010113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4010546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4010891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4011204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4011665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4011993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4012410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4012755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4013082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4013373Z skip: Need at least 2 CUDA devices (1.750s) 2022-12-01T10:10:59.4013511Z 2022-12-01T10:10:59.4013697Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4013936Z Ran 1 test in 1.750s 2022-12-01T10:10:59.4014050Z 2022-12-01T10:10:59.4014123Z OK (skipped=1) 2022-12-01T10:10:59.4014232Z 2022-12-01T10:10:59.4014302Z Generating XML reports... 2022-12-01T10:10:59.4014768Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100854.xml 2022-12-01T10:10:59.4015269Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4015733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4016050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4016479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4016883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4017050Z 2022-12-01T10:10:59.4017124Z Running tests... 2022-12-01T10:10:59.4017416Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4017756Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4018246Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6497 2022-12-01T10:10:59.4018636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6498 2022-12-01T10:10:59.4019095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4019463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4019895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4020231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4020554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4021020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4021350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4021763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4022108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4022432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4022710Z skip: Need at least 2 CUDA devices (1.745s) 2022-12-01T10:10:59.4022846Z 2022-12-01T10:10:59.4023044Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4023286Z Ran 1 test in 1.745s 2022-12-01T10:10:59.4023399Z 2022-12-01T10:10:59.4023470Z OK (skipped=1) 2022-12-01T10:10:59.4023564Z 2022-12-01T10:10:59.4023649Z Generating XML reports... 2022-12-01T10:10:59.4024122Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100857.xml 2022-12-01T10:10:59.4024628Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4025078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4025407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4025837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4026185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4026334Z 2022-12-01T10:10:59.4026407Z Running tests... 2022-12-01T10:10:59.4026707Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4027059Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4027494Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6530 2022-12-01T10:10:59.4027901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6531 2022-12-01T10:10:59.4028350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4028683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4029098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4029444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4029767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4030223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4030571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4030999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4031343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4031653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4031943Z skip: Need at least 2 CUDA devices (1.742s) 2022-12-01T10:10:59.4032120Z 2022-12-01T10:10:59.4032319Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4032560Z Ran 1 test in 1.743s 2022-12-01T10:10:59.4032660Z 2022-12-01T10:10:59.4032732Z OK (skipped=1) 2022-12-01T10:10:59.4032839Z 2022-12-01T10:10:59.4032925Z Generating XML reports... 2022-12-01T10:10:59.4033391Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100900.xml 2022-12-01T10:10:59.4033881Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4034344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4034674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4035101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4035435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4035601Z 2022-12-01T10:10:59.4035673Z Running tests... 2022-12-01T10:10:59.4035977Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4036314Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4036766Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6563 2022-12-01T10:10:59.4037171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6564 2022-12-01T10:10:59.4037628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4037948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4038497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4038848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4039175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4039620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4039954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4040383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4040713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4041039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4041334Z skip: Need at least 2 CUDA devices (1.742s) 2022-12-01T10:10:59.4041468Z 2022-12-01T10:10:59.4041664Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4041893Z Ran 1 test in 1.743s 2022-12-01T10:10:59.4042004Z 2022-12-01T10:10:59.4042078Z OK (skipped=1) 2022-12-01T10:10:59.4042186Z 2022-12-01T10:10:59.4042271Z Generating XML reports... 2022-12-01T10:10:59.4042724Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100903.xml 2022-12-01T10:10:59.4043228Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4043743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4044075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4044490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4044840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4045061Z 2022-12-01T10:10:59.4045136Z Running tests... 2022-12-01T10:10:59.4045428Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4045783Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4046206Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6596 2022-12-01T10:10:59.4046591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6597 2022-12-01T10:10:59.4047037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4047371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4047803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4048152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4048464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4048927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4049255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4049666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4050013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4050338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4050628Z skip: Need at least 2 CUDA devices (1.744s) 2022-12-01T10:10:59.4050752Z 2022-12-01T10:10:59.4050951Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4051191Z Ran 1 test in 1.744s 2022-12-01T10:10:59.4051306Z 2022-12-01T10:10:59.4051379Z OK (skipped=1) 2022-12-01T10:10:59.4051486Z 2022-12-01T10:10:59.4051557Z Generating XML reports... 2022-12-01T10:10:59.4052027Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100907.xml 2022-12-01T10:10:59.4052531Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4052992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4053310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4053738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4054085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4054250Z 2022-12-01T10:10:59.4054309Z Running tests... 2022-12-01T10:10:59.4054611Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4054968Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4055391Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6629 2022-12-01T10:10:59.4055760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6630 2022-12-01T10:10:59.4056248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4056582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4057079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4057414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4057738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4058235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4058550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4058981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4059331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4059658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4059935Z skip: Need at least 2 CUDA devices (1.744s) 2022-12-01T10:10:59.4060070Z 2022-12-01T10:10:59.4060268Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4060509Z Ran 1 test in 1.744s 2022-12-01T10:10:59.4060623Z 2022-12-01T10:10:59.4060682Z OK (skipped=1) 2022-12-01T10:10:59.4060790Z 2022-12-01T10:10:59.4060878Z Generating XML reports... 2022-12-01T10:10:59.4061345Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100910.xml 2022-12-01T10:10:59.4061846Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4062296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4062627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4063061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4063394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4063557Z 2022-12-01T10:10:59.4063629Z Running tests... 2022-12-01T10:10:59.4063932Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4064320Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4064560Z 2022-12-01T10:10:59.4064742Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4064978Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4065091Z 2022-12-01T10:10:59.4065164Z OK (skipped=1) 2022-12-01T10:10:59.4065271Z 2022-12-01T10:10:59.4065342Z Generating XML reports... 2022-12-01T10:10:59.4065811Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100913.xml 2022-12-01T10:10:59.4066313Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4066587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4066711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4066991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4067115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4067134Z 2022-12-01T10:10:59.4067193Z Running tests... 2022-12-01T10:10:59.4067389Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4067626Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4067663Z 2022-12-01T10:10:59.4067860Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4067936Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4067941Z 2022-12-01T10:10:59.4068013Z OK (skipped=1) 2022-12-01T10:10:59.4068018Z 2022-12-01T10:10:59.4068103Z Generating XML reports... 2022-12-01T10:10:59.4068448Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100915.xml 2022-12-01T10:10:59.4068698Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4068974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4069098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4069383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4069523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4069527Z 2022-12-01T10:10:59.4069604Z Running tests... 2022-12-01T10:10:59.4069800Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4070037Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4070046Z 2022-12-01T10:10:59.4070240Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4070302Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4070306Z 2022-12-01T10:10:59.4070379Z OK (skipped=1) 2022-12-01T10:10:59.4070384Z 2022-12-01T10:10:59.4070468Z Generating XML reports... 2022-12-01T10:10:59.4070812Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100916.xml 2022-12-01T10:10:59.4071045Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4071318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4071443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4071727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4071864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4071872Z 2022-12-01T10:10:59.4071932Z Running tests... 2022-12-01T10:10:59.4072127Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4072298Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4072515Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4072520Z 2022-12-01T10:10:59.4072714Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4072789Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4072793Z 2022-12-01T10:10:59.4072865Z OK (skipped=1) 2022-12-01T10:10:59.4072869Z 2022-12-01T10:10:59.4072954Z Generating XML reports... 2022-12-01T10:10:59.4073305Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100917.xml 2022-12-01T10:10:59.4073526Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4073798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4073923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4074204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4074368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4074373Z 2022-12-01T10:10:59.4074446Z Running tests... 2022-12-01T10:10:59.4074642Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4074808Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4075021Z This unit test verifies whether the Future object is passed properly using nccl backend. ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4075070Z 2022-12-01T10:10:59.4075250Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4075325Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4075330Z 2022-12-01T10:10:59.4075402Z OK (skipped=1) 2022-12-01T10:10:59.4075406Z 2022-12-01T10:10:59.4075490Z Generating XML reports... 2022-12-01T10:10:59.4075838Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100919.xml 2022-12-01T10:10:59.4076073Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4076349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4076472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4076738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4076877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4076881Z 2022-12-01T10:10:59.4076953Z Running tests... 2022-12-01T10:10:59.4077146Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4077368Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4077374Z 2022-12-01T10:10:59.4077569Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4077646Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4077651Z 2022-12-01T10:10:59.4077723Z OK (skipped=1) 2022-12-01T10:10:59.4077728Z 2022-12-01T10:10:59.4077812Z Generating XML reports... 2022-12-01T10:10:59.4078142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100920.xml 2022-12-01T10:10:59.4078474Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4078750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4078876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4079157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4079294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4079302Z 2022-12-01T10:10:59.4079378Z Running tests... 2022-12-01T10:10:59.4079580Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4079793Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4079798Z 2022-12-01T10:10:59.4079978Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4080057Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4080062Z 2022-12-01T10:10:59.4080133Z OK (skipped=1) 2022-12-01T10:10:59.4080137Z 2022-12-01T10:10:59.4080222Z Generating XML reports... 2022-12-01T10:10:59.4080568Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100922.xml 2022-12-01T10:10:59.4080799Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4081117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4081243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4081526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4081649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4081654Z 2022-12-01T10:10:59.4081727Z Running tests... 2022-12-01T10:10:59.4081955Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4082172Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4082176Z 2022-12-01T10:10:59.4082367Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4082444Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4082449Z 2022-12-01T10:10:59.4082523Z OK (skipped=1) 2022-12-01T10:10:59.4082527Z 2022-12-01T10:10:59.4082614Z Generating XML reports... 2022-12-01T10:10:59.4082942Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100923.xml 2022-12-01T10:10:59.4083172Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4083444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4083572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4083853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4083990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4083996Z 2022-12-01T10:10:59.4084068Z Running tests... 2022-12-01T10:10:59.4084261Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4084482Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4084487Z 2022-12-01T10:10:59.4084665Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4084742Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4084747Z 2022-12-01T10:10:59.4084820Z OK (skipped=1) 2022-12-01T10:10:59.4084824Z 2022-12-01T10:10:59.4084909Z Generating XML reports... 2022-12-01T10:10:59.4085257Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100924.xml 2022-12-01T10:10:59.4085488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4085761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4085886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4086169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4086292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4086296Z 2022-12-01T10:10:59.4086371Z Running tests... 2022-12-01T10:10:59.4086564Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4086789Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4086796Z 2022-12-01T10:10:59.4086991Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4087066Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4087071Z 2022-12-01T10:10:59.4087142Z OK (skipped=1) 2022-12-01T10:10:59.4087147Z 2022-12-01T10:10:59.4087232Z Generating XML reports... 2022-12-01T10:10:59.4087629Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100926.xml 2022-12-01T10:10:59.4087847Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4088122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4088248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4088529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4088699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4088704Z 2022-12-01T10:10:59.4088776Z Running tests... 2022-12-01T10:10:59.4088972Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4089180Z test_failure_recovery (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-12-01T10:10:59.4089185Z 2022-12-01T10:10:59.4089382Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4089445Z Ran 1 test in 0.002s 2022-12-01T10:10:59.4089449Z 2022-12-01T10:10:59.4089522Z OK (skipped=1) 2022-12-01T10:10:59.4089526Z 2022-12-01T10:10:59.4089610Z Generating XML reports... 2022-12-01T10:10:59.4089956Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100927.xml 2022-12-01T10:10:59.4090189Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4090463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4090587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4090869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4090993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4091011Z 2022-12-01T10:10:59.4091070Z Running tests... 2022-12-01T10:10:59.4091265Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4091502Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4091509Z 2022-12-01T10:10:59.4091700Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4091778Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4091783Z 2022-12-01T10:10:59.4091855Z OK (skipped=1) 2022-12-01T10:10:59.4091860Z 2022-12-01T10:10:59.4091945Z Generating XML reports... 2022-12-01T10:10:59.4092290Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100928.xml 2022-12-01T10:10:59.4092504Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4092780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4092904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4093188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4093329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4093336Z 2022-12-01T10:10:59.4093409Z Running tests... 2022-12-01T10:10:59.4093606Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4093837Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4093842Z 2022-12-01T10:10:59.4094035Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4094098Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4094103Z 2022-12-01T10:10:59.4094202Z OK (skipped=1) 2022-12-01T10:10:59.4094207Z 2022-12-01T10:10:59.4094292Z Generating XML reports... 2022-12-01T10:10:59.4094643Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100930.xml 2022-12-01T10:10:59.4094875Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4095150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4095305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4095585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4095725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4095730Z 2022-12-01T10:10:59.4095790Z Running tests... 2022-12-01T10:10:59.4095987Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4096221Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4096226Z 2022-12-01T10:10:59.4096419Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4096494Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4096499Z 2022-12-01T10:10:59.4096570Z OK (skipped=1) 2022-12-01T10:10:59.4096576Z 2022-12-01T10:10:59.4096659Z Generating XML reports... 2022-12-01T10:10:59.4097063Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100931.xml 2022-12-01T10:10:59.4097300Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4097561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4097685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4097963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4098100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4098104Z 2022-12-01T10:10:59.4098176Z Running tests... 2022-12-01T10:10:59.4098372Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4098622Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4098627Z 2022-12-01T10:10:59.4098822Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4098897Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4098901Z 2022-12-01T10:10:59.4098959Z OK (skipped=1) 2022-12-01T10:10:59.4098964Z 2022-12-01T10:10:59.4099049Z Generating XML reports... 2022-12-01T10:10:59.4099397Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100933.xml 2022-12-01T10:10:59.4099626Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4099898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4100023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4100306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4100443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4100448Z 2022-12-01T10:10:59.4100507Z Running tests... 2022-12-01T10:10:59.4100701Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4100982Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4100989Z 2022-12-01T10:10:59.4101187Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4101264Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4101269Z 2022-12-01T10:10:59.4101342Z OK (skipped=1) 2022-12-01T10:10:59.4101347Z 2022-12-01T10:10:59.4101434Z Generating XML reports... 2022-12-01T10:10:59.4101780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100934.xml 2022-12-01T10:10:59.4102041Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4102301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4102429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4102714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4102850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4102854Z 2022-12-01T10:10:59.4102928Z Running tests... 2022-12-01T10:10:59.4103124Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4103367Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4103374Z 2022-12-01T10:10:59.4103567Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4103643Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4103648Z 2022-12-01T10:10:59.4103707Z OK (skipped=1) 2022-12-01T10:10:59.4103725Z 2022-12-01T10:10:59.4103796Z Generating XML reports... 2022-12-01T10:10:59.4104142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100935.xml 2022-12-01T10:10:59.4104373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4104649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4104773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4105055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4105193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4105197Z 2022-12-01T10:10:59.4105269Z Running tests... 2022-12-01T10:10:59.4105450Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4105642Z test_fp16 (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4105648Z 2022-12-01T10:10:59.4105840Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4105916Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4105920Z 2022-12-01T10:10:59.4105992Z OK (skipped=1) 2022-12-01T10:10:59.4105996Z 2022-12-01T10:10:59.4106080Z Generating XML reports... 2022-12-01T10:10:59.4106423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100937.xml 2022-12-01T10:10:59.4106655Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4106930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4107041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4107319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4107459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4107464Z 2022-12-01T10:10:59.4107566Z Running tests... 2022-12-01T10:10:59.4107764Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4107986Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4107991Z 2022-12-01T10:10:59.4108185Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4108288Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4108293Z 2022-12-01T10:10:59.4108351Z OK (skipped=1) 2022-12-01T10:10:59.4108370Z 2022-12-01T10:10:59.4108441Z Generating XML reports... 2022-12-01T10:10:59.4108787Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100938.xml 2022-12-01T10:10:59.4109013Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4109287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4109412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4109691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4109826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4109830Z 2022-12-01T10:10:59.4109903Z Running tests... 2022-12-01T10:10:59.4110085Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4110302Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4110307Z 2022-12-01T10:10:59.4110501Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4110576Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4110581Z 2022-12-01T10:10:59.4110652Z OK (skipped=1) 2022-12-01T10:10:59.4110657Z 2022-12-01T10:10:59.4110744Z Generating XML reports... 2022-12-01T10:10:59.4111089Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100940.xml 2022-12-01T10:10:59.4111324Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4111597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4111712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4111996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4112134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4112138Z 2022-12-01T10:10:59.4112211Z Running tests... 2022-12-01T10:10:59.4112405Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4112612Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4112618Z 2022-12-01T10:10:59.4112814Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4112892Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4112897Z 2022-12-01T10:10:59.4112969Z OK (skipped=1) 2022-12-01T10:10:59.4112973Z 2022-12-01T10:10:59.4113044Z Generating XML reports... 2022-12-01T10:10:59.4113392Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100941.xml 2022-12-01T10:10:59.4113623Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4113897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4114021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4114330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4114467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4114472Z 2022-12-01T10:10:59.4114545Z Running tests... 2022-12-01T10:10:59.4114725Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4114969Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4115002Z 2022-12-01T10:10:59.4115198Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4115275Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4115280Z 2022-12-01T10:10:59.4115351Z OK (skipped=1) 2022-12-01T10:10:59.4115355Z 2022-12-01T10:10:59.4115440Z Generating XML reports... 2022-12-01T10:10:59.4115791Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100942.xml 2022-12-01T10:10:59.4116023Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4116297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4116409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4116692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4116831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4116836Z 2022-12-01T10:10:59.4116909Z Running tests... 2022-12-01T10:10:59.4117105Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4117324Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4117329Z 2022-12-01T10:10:59.4117525Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4117600Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4117604Z 2022-12-01T10:10:59.4117676Z OK (skipped=1) 2022-12-01T10:10:59.4117681Z 2022-12-01T10:10:59.4117752Z Generating XML reports... 2022-12-01T10:10:59.4118096Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100944.xml 2022-12-01T10:10:59.4118444Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4118720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4118847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4119129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4119268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4119273Z 2022-12-01T10:10:59.4119347Z Running tests... 2022-12-01T10:10:59.4119541Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4119773Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6892 2022-12-01T10:10:59.4119930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6893 2022-12-01T10:10:59.4120206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4120329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4120614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4120753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4120966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4121241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4121365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4121633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4121803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4121968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4122387Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4122797Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4123211Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4123622Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4124029Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4124435Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4124844Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4125247Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4125655Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4126086Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4126484Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4126917Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:10:59.4126989Z ok (1.743s) 2022-12-01T10:10:59.4126994Z 2022-12-01T10:10:59.4127194Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4127269Z Ran 1 test in 1.743s 2022-12-01T10:10:59.4127275Z 2022-12-01T10:10:59.4127336Z OK 2022-12-01T10:10:59.4127341Z 2022-12-01T10:10:59.4127427Z Generating XML reports... 2022-12-01T10:10:59.4127762Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100945.xml 2022-12-01T10:10:59.4127998Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4128269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4128395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4128679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4128819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4128824Z 2022-12-01T10:10:59.4128898Z Running tests... 2022-12-01T10:10:59.4129095Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4129321Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4129329Z 2022-12-01T10:10:59.4129511Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4129586Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4129591Z 2022-12-01T10:10:59.4129663Z OK (skipped=1) 2022-12-01T10:10:59.4129667Z 2022-12-01T10:10:59.4129754Z Generating XML reports... 2022-12-01T10:10:59.4130102Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100948.xml 2022-12-01T10:10:59.4130336Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4130612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4130737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4131021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4131144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4131151Z 2022-12-01T10:10:59.4131225Z Running tests... 2022-12-01T10:10:59.4131419Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4131658Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4131664Z 2022-12-01T10:10:59.4131858Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4131973Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4131978Z 2022-12-01T10:10:59.4132052Z OK (skipped=1) 2022-12-01T10:10:59.4132057Z 2022-12-01T10:10:59.4132141Z Generating XML reports... 2022-12-01T10:10:59.4132484Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100950.xml 2022-12-01T10:10:59.4132702Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4133014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4133139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4133420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4133555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4133561Z 2022-12-01T10:10:59.4133636Z Running tests... 2022-12-01T10:10:59.4133830Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4134070Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4134075Z 2022-12-01T10:10:59.4134269Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4134330Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4134337Z 2022-12-01T10:10:59.4134410Z OK (skipped=1) 2022-12-01T10:10:59.4134414Z 2022-12-01T10:10:59.4134500Z Generating XML reports... 2022-12-01T10:10:59.4134842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100951.xml 2022-12-01T10:10:59.4135073Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4135348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4135473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4135755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4135878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4135897Z 2022-12-01T10:10:59.4135955Z Running tests... 2022-12-01T10:10:59.4136151Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4136395Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4136400Z 2022-12-01T10:10:59.4136592Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4136668Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4136672Z 2022-12-01T10:10:59.4136745Z OK (skipped=1) 2022-12-01T10:10:59.4136749Z 2022-12-01T10:10:59.4136895Z Generating XML reports... 2022-12-01T10:10:59.4137245Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100952.xml 2022-12-01T10:10:59.4137461Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4137736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4137863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4138141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4138278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4138283Z 2022-12-01T10:10:59.4138356Z Running tests... 2022-12-01T10:10:59.4138553Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4138804Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4138810Z 2022-12-01T10:10:59.4139002Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4139064Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4139085Z 2022-12-01T10:10:59.4139143Z OK (skipped=1) 2022-12-01T10:10:59.4139147Z 2022-12-01T10:10:59.4139230Z Generating XML reports... 2022-12-01T10:10:59.4139607Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100954.xml 2022-12-01T10:10:59.4139839Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4140111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4140236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4140518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4140655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4140660Z 2022-12-01T10:10:59.4140720Z Running tests... 2022-12-01T10:10:59.4140919Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4141136Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4141144Z 2022-12-01T10:10:59.4141337Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4141413Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4141418Z 2022-12-01T10:10:59.4141490Z OK (skipped=1) 2022-12-01T10:10:59.4141494Z 2022-12-01T10:10:59.4141579Z Generating XML reports... 2022-12-01T10:10:59.4141924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100955.xml 2022-12-01T10:10:59.4142154Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4142412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4142537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4142818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4142958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4142963Z 2022-12-01T10:10:59.4143035Z Running tests... 2022-12-01T10:10:59.4143229Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4143461Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4143466Z 2022-12-01T10:10:59.4143662Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4143736Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4143742Z 2022-12-01T10:10:59.4143800Z OK (skipped=1) 2022-12-01T10:10:59.4143805Z 2022-12-01T10:10:59.4143891Z Generating XML reports... 2022-12-01T10:10:59.4144238Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100957.xml 2022-12-01T10:10:59.4144472Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4144746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4144870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4145155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4145317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4145323Z 2022-12-01T10:10:59.4145382Z Running tests... 2022-12-01T10:10:59.4145577Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4145815Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4145819Z 2022-12-01T10:10:59.4146012Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4146117Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4146122Z 2022-12-01T10:10:59.4146195Z OK (skipped=1) 2022-12-01T10:10:59.4146199Z 2022-12-01T10:10:59.4146283Z Generating XML reports... 2022-12-01T10:10:59.4146628Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100958.xml 2022-12-01T10:10:59.4146858Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4147118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4147244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4147523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4147658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4147665Z 2022-12-01T10:10:59.4147740Z Running tests... 2022-12-01T10:10:59.4147938Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4148180Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4148186Z 2022-12-01T10:10:59.4148379Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4148453Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4148460Z 2022-12-01T10:10:59.4148519Z OK (skipped=1) 2022-12-01T10:10:59.4148538Z 2022-12-01T10:10:59.4148608Z Generating XML reports... 2022-12-01T10:10:59.4148948Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100959.xml 2022-12-01T10:10:59.4149178Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4149449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4149577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4149856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4149992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4149997Z 2022-12-01T10:10:59.4150069Z Running tests... 2022-12-01T10:10:59.4150254Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4150498Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4150502Z 2022-12-01T10:10:59.4150696Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4150771Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4150775Z 2022-12-01T10:10:59.4150855Z OK (skipped=1) 2022-12-01T10:10:59.4150859Z 2022-12-01T10:10:59.4150944Z Generating XML reports... 2022-12-01T10:10:59.4151289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101001.xml 2022-12-01T10:10:59.4151517Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4151791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4151929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4152213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4152349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4152354Z 2022-12-01T10:10:59.4152426Z Running tests... 2022-12-01T10:10:59.4152620Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4152870Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4152876Z 2022-12-01T10:10:59.4153069Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4153144Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4153149Z 2022-12-01T10:10:59.4153219Z OK (skipped=1) 2022-12-01T10:10:59.4153224Z 2022-12-01T10:10:59.4153294Z Generating XML reports... 2022-12-01T10:10:59.4153641Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101002.xml 2022-12-01T10:10:59.4153871Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4154142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4154266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4154553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4154692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4154696Z 2022-12-01T10:10:59.4154772Z Running tests... 2022-12-01T10:10:59.4154950Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4155081Z test_no_grad (__main__.DistributedDataParallelTest) 2022-12-01T10:10:59.4155280Z Note: this test can be sped up by only running it on a CPU module ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-12-01T10:10:59.4155285Z 2022-12-01T10:10:59.4155477Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4155551Z Ran 1 test in 0.002s 2022-12-01T10:10:59.4155556Z 2022-12-01T10:10:59.4155628Z OK (skipped=1) 2022-12-01T10:10:59.4155633Z 2022-12-01T10:10:59.4155717Z Generating XML reports... 2022-12-01T10:10:59.4156062Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101004.xml 2022-12-01T10:10:59.4156292Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4156550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4195637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4196117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4196261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4196269Z 2022-12-01T10:10:59.4196330Z Running tests... 2022-12-01T10:10:59.4196532Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4196753Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4196766Z 2022-12-01T10:10:59.4196961Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4197036Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4197041Z 2022-12-01T10:10:59.4197111Z OK (skipped=1) 2022-12-01T10:10:59.4197115Z 2022-12-01T10:10:59.4197200Z Generating XML reports... 2022-12-01T10:10:59.4197673Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101005.xml 2022-12-01T10:10:59.4197910Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4198174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4198424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4198705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4198894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4198899Z 2022-12-01T10:10:59.4198970Z Running tests... 2022-12-01T10:10:59.4199166Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4199369Z test_pass_default_pg (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4199375Z 2022-12-01T10:10:59.4199569Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4199630Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4199648Z 2022-12-01T10:10:59.4199707Z OK (skipped=1) 2022-12-01T10:10:59.4199711Z 2022-12-01T10:10:59.4199795Z Generating XML reports... 2022-12-01T10:10:59.4200140Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101006.xml 2022-12-01T10:10:59.4200368Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4200637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4200763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4201047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4201183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4201188Z 2022-12-01T10:10:59.4201247Z Running tests... 2022-12-01T10:10:59.4201445Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4201662Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4201667Z 2022-12-01T10:10:59.4201856Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4201931Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4201935Z 2022-12-01T10:10:59.4202002Z OK (skipped=1) 2022-12-01T10:10:59.4202006Z 2022-12-01T10:10:59.4202086Z Generating XML reports... 2022-12-01T10:10:59.4202426Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101008.xml 2022-12-01T10:10:59.4202652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4202914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4203036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4203312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4203444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4203451Z 2022-12-01T10:10:59.4203521Z Running tests... 2022-12-01T10:10:59.4203713Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4203939Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4203945Z 2022-12-01T10:10:59.4204134Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4204205Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4204209Z 2022-12-01T10:10:59.4204302Z OK (skipped=1) 2022-12-01T10:10:59.4204307Z 2022-12-01T10:10:59.4204389Z Generating XML reports... 2022-12-01T10:10:59.4204731Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101009.xml 2022-12-01T10:10:59.4204955Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4205229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4205381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4205659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4205791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4205796Z 2022-12-01T10:10:59.4205865Z Running tests... 2022-12-01T10:10:59.4206050Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4206291Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7085 2022-12-01T10:10:59.4206445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7086 2022-12-01T10:10:59.4206718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4206841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4207115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4207247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4207410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4207666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4207785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4208056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4208186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4208347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4208451Z skip: Need at least 2 CUDA devices (1.749s) 2022-12-01T10:10:59.4208456Z 2022-12-01T10:10:59.4208653Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4208725Z Ran 1 test in 1.749s 2022-12-01T10:10:59.4208730Z 2022-12-01T10:10:59.4208802Z OK (skipped=1) 2022-12-01T10:10:59.4208806Z 2022-12-01T10:10:59.4208876Z Generating XML reports... 2022-12-01T10:10:59.4209226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101010.xml 2022-12-01T10:10:59.4209448Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4209714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4209833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4210109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4210242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4210247Z 2022-12-01T10:10:59.4210311Z Running tests... 2022-12-01T10:10:59.4210493Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4210767Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7118 2022-12-01T10:10:59.4210924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7119 2022-12-01T10:10:59.4211191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4211310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4211583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4211743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4211904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:10:59.4212170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4212281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4212554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4212687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4212845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:10:59.4212946Z skip: Need at least 2 CUDA devices (1.744s) 2022-12-01T10:10:59.4212951Z 2022-12-01T10:10:59.4213143Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4213215Z Ran 1 test in 1.744s 2022-12-01T10:10:59.4213220Z 2022-12-01T10:10:59.4213289Z OK (skipped=1) 2022-12-01T10:10:59.4213293Z 2022-12-01T10:10:59.4213374Z Generating XML reports... 2022-12-01T10:10:59.4213711Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101014.xml 2022-12-01T10:10:59.4213933Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4214203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4214319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4214593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4214720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4214727Z 2022-12-01T10:10:59.4214794Z Running tests... 2022-12-01T10:10:59.4214981Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4215177Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4215190Z 2022-12-01T10:10:59.4215370Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4215436Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4215441Z 2022-12-01T10:10:59.4215508Z OK (skipped=1) 2022-12-01T10:10:59.4215513Z 2022-12-01T10:10:59.4215594Z Generating XML reports... 2022-12-01T10:10:59.4215911Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101017.xml 2022-12-01T10:10:59.4216135Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4216402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4216519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4216870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4217007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4217012Z 2022-12-01T10:10:59.4217076Z Running tests... 2022-12-01T10:10:59.4217327Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4217531Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4217536Z 2022-12-01T10:10:59.4217720Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4217789Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4217794Z 2022-12-01T10:10:59.4217858Z OK (skipped=1) 2022-12-01T10:10:59.4217863Z 2022-12-01T10:10:59.4217973Z Generating XML reports... 2022-12-01T10:10:59.4218287Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101018.xml 2022-12-01T10:10:59.4218508Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4218777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4218898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4219175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4219309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4219314Z 2022-12-01T10:10:59.4219379Z Running tests... 2022-12-01T10:10:59.4219568Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4219761Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4219769Z 2022-12-01T10:10:59.4219947Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4220015Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4220020Z 2022-12-01T10:10:59.4220084Z OK (skipped=1) 2022-12-01T10:10:59.4220088Z 2022-12-01T10:10:59.4220165Z Generating XML reports... 2022-12-01T10:10:59.4220487Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101020.xml 2022-12-01T10:10:59.4220709Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4220977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4221092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4221356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4221486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4221491Z 2022-12-01T10:10:59.4221556Z Running tests... 2022-12-01T10:10:59.4221745Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4221945Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4221950Z 2022-12-01T10:10:59.4222256Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4222325Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4222330Z 2022-12-01T10:10:59.4222395Z OK (skipped=1) 2022-12-01T10:10:59.4222399Z 2022-12-01T10:10:59.4222480Z Generating XML reports... 2022-12-01T10:10:59.4222793Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101021.xml 2022-12-01T10:10:59.4223020Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4223288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4223404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4223681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4223838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4223843Z 2022-12-01T10:10:59.4223910Z Running tests... 2022-12-01T10:10:59.4224097Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4224303Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4224309Z 2022-12-01T10:10:59.4224488Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4224586Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4224591Z 2022-12-01T10:10:59.4224654Z OK (skipped=1) 2022-12-01T10:10:59.4224658Z 2022-12-01T10:10:59.4224737Z Generating XML reports... 2022-12-01T10:10:59.4225054Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101022.xml 2022-12-01T10:10:59.4225277Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4225543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4225662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4225937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4226059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4226066Z 2022-12-01T10:10:59.4226132Z Running tests... 2022-12-01T10:10:59.4226319Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4226520Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4226525Z 2022-12-01T10:10:59.4226709Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4226779Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4226784Z 2022-12-01T10:10:59.4226850Z OK (skipped=1) 2022-12-01T10:10:59.4226855Z 2022-12-01T10:10:59.4226932Z Generating XML reports... 2022-12-01T10:10:59.4227241Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101024.xml 2022-12-01T10:10:59.4227464Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4227726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4227844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4228114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4228241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4228246Z 2022-12-01T10:10:59.4228315Z Running tests... 2022-12-01T10:10:59.4228506Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4228704Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4228709Z 2022-12-01T10:10:59.4228893Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4228954Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4228959Z 2022-12-01T10:10:59.4229022Z OK (skipped=1) 2022-12-01T10:10:59.4229026Z 2022-12-01T10:10:59.4229105Z Generating XML reports... 2022-12-01T10:10:59.4229425Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101025.xml 2022-12-01T10:10:59.4229652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4229916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4230031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4230330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4230454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4230458Z 2022-12-01T10:10:59.4230523Z Running tests... 2022-12-01T10:10:59.4230709Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4230899Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4230932Z 2022-12-01T10:10:59.4231113Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4231179Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4231184Z 2022-12-01T10:10:59.4231247Z OK (skipped=1) 2022-12-01T10:10:59.4231252Z 2022-12-01T10:10:59.4231329Z Generating XML reports... 2022-12-01T10:10:59.4231652Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101027.xml 2022-12-01T10:10:59.4231868Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4232135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4232249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4232526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4232655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4232660Z 2022-12-01T10:10:59.4232724Z Running tests... 2022-12-01T10:10:59.4232916Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4233097Z test_nccl_timeout (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4233102Z 2022-12-01T10:10:59.4233293Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4233355Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4233359Z 2022-12-01T10:10:59.4233428Z OK (skipped=1) 2022-12-01T10:10:59.4233433Z 2022-12-01T10:10:59.4233512Z Generating XML reports... 2022-12-01T10:10:59.4233836Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101028.xml 2022-12-01T10:10:59.4234070Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4234339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4234459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4234739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4234863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4234876Z 2022-12-01T10:10:59.4234935Z Running tests... 2022-12-01T10:10:59.4235126Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4235374Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4235379Z 2022-12-01T10:10:59.4235569Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4235645Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4235649Z 2022-12-01T10:10:59.4235718Z OK (skipped=1) 2022-12-01T10:10:59.4235722Z 2022-12-01T10:10:59.4235802Z Generating XML reports... 2022-12-01T10:10:59.4236223Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221201101029.xml 2022-12-01T10:10:59.4236442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4236749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4236871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4237151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4237283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4237315Z 2022-12-01T10:10:59.4237387Z Running tests... 2022-12-01T10:10:59.4237581Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4237778Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4237783Z 2022-12-01T10:10:59.4237969Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4238031Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4238035Z 2022-12-01T10:10:59.4238108Z OK (skipped=1) 2022-12-01T10:10:59.4238113Z 2022-12-01T10:10:59.4238306Z Generating XML reports... 2022-12-01T10:10:59.4238645Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221201101031.xml 2022-12-01T10:10:59.4238871Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4239140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4239262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4239541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4239674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4239679Z 2022-12-01T10:10:59.4239739Z Running tests... 2022-12-01T10:10:59.4239931Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4240127Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4240131Z 2022-12-01T10:10:59.4240323Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4240395Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4240399Z 2022-12-01T10:10:59.4240468Z OK (skipped=1) 2022-12-01T10:10:59.4240473Z 2022-12-01T10:10:59.4240556Z Generating XML reports... 2022-12-01T10:10:59.4240879Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101032.xml 2022-12-01T10:10:59.4241092Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4241362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4241483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4241764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4241897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4241903Z 2022-12-01T10:10:59.4241972Z Running tests... 2022-12-01T10:10:59.4242165Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4242362Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4242369Z 2022-12-01T10:10:59.4242558Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4242619Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4242623Z 2022-12-01T10:10:59.4242692Z OK (skipped=1) 2022-12-01T10:10:59.4242696Z 2022-12-01T10:10:59.4242777Z Generating XML reports... 2022-12-01T10:10:59.4243145Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101034.xml 2022-12-01T10:10:59.4243375Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4243646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4243767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4244047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4244218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4244223Z 2022-12-01T10:10:59.4244283Z Running tests... 2022-12-01T10:10:59.4244478Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4244670Z test_allgather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4244675Z 2022-12-01T10:10:59.4244868Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4244940Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4244944Z 2022-12-01T10:10:59.4245015Z OK (skipped=1) 2022-12-01T10:10:59.4245020Z 2022-12-01T10:10:59.4245102Z Generating XML reports... 2022-12-01T10:10:59.4245423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101035.xml 2022-12-01T10:10:59.4245639Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4245911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4246033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4246310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4246442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4246450Z 2022-12-01T10:10:59.4246517Z Running tests... 2022-12-01T10:10:59.4246712Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4246903Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-12-01T10:10:59.4246907Z 2022-12-01T10:10:59.4247096Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4247159Z Ran 1 test in 0.003s 2022-12-01T10:10:59.4247175Z 2022-12-01T10:10:59.4247234Z OK (skipped=1) 2022-12-01T10:10:59.4247238Z 2022-12-01T10:10:59.4247318Z Generating XML reports... 2022-12-01T10:10:59.4247638Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101036.xml 2022-12-01T10:10:59.4247864Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4248138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4248259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4248535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4248670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4248675Z 2022-12-01T10:10:59.4248734Z Running tests... 2022-12-01T10:10:59.4248928Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4249128Z test_barrier (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4249133Z 2022-12-01T10:10:59.4249325Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4249398Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4249402Z 2022-12-01T10:10:59.4249470Z OK (skipped=1) 2022-12-01T10:10:59.4249474Z 2022-12-01T10:10:59.4249580Z Generating XML reports... 2022-12-01T10:10:59.4249903Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101038.xml 2022-12-01T10:10:59.4250130Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4250389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4250511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4250817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4250950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4250955Z 2022-12-01T10:10:59.4251026Z Running tests... 2022-12-01T10:10:59.4251218Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4251409Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4251415Z 2022-12-01T10:10:59.4251606Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4251668Z Ran 1 test in 0.002s 2022-12-01T10:10:59.4251684Z 2022-12-01T10:10:59.4251743Z OK (skipped=1) 2022-12-01T10:10:59.4251747Z 2022-12-01T10:10:59.4251829Z Generating XML reports... 2022-12-01T10:10:59.4252147Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101039.xml 2022-12-01T10:10:59.4252375Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4252646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4252767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4253045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4253183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4253188Z 2022-12-01T10:10:59.4253248Z Running tests... 2022-12-01T10:10:59.4253441Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4253626Z test_empty_tensors (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4253634Z 2022-12-01T10:10:59.4253822Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4253893Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4253898Z 2022-12-01T10:10:59.4253968Z OK (skipped=1) 2022-12-01T10:10:59.4253973Z 2022-12-01T10:10:59.4254055Z Generating XML reports... 2022-12-01T10:10:59.4254374Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101040.xml 2022-12-01T10:10:59.4254601Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4254860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4254978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4255253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4255386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4255393Z 2022-12-01T10:10:59.4255464Z Running tests... 2022-12-01T10:10:59.4255658Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4255843Z test_gather_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-12-01T10:10:59.4255849Z 2022-12-01T10:10:59.4256040Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4256130Z Ran 1 test in 0.002s 2022-12-01T10:10:59.4256148Z 2022-12-01T10:10:59.4256207Z OK (skipped=1) 2022-12-01T10:10:59.4256211Z 2022-12-01T10:10:59.4256292Z Generating XML reports... 2022-12-01T10:10:59.4256612Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101042.xml 2022-12-01T10:10:59.4256899Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4257174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4257331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4257610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4257746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4257751Z 2022-12-01T10:10:59.4257812Z Running tests... 2022-12-01T10:10:59.4258009Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4258196Z test_gather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4258201Z 2022-12-01T10:10:59.4258385Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4258457Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4258462Z 2022-12-01T10:10:59.4258539Z OK (skipped=1) 2022-12-01T10:10:59.4258546Z 2022-12-01T10:10:59.4258630Z Generating XML reports... 2022-12-01T10:10:59.4258948Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101043.xml 2022-12-01T10:10:59.4259174Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4259436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4259561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4259839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4259975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4259979Z 2022-12-01T10:10:59.4260050Z Running tests... 2022-12-01T10:10:59.4260246Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4260436Z test_gather_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-12-01T10:10:59.4260441Z 2022-12-01T10:10:59.4260627Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4260687Z Ran 1 test in 0.002s 2022-12-01T10:10:59.4260703Z 2022-12-01T10:10:59.4260761Z OK (skipped=1) 2022-12-01T10:10:59.4260765Z 2022-12-01T10:10:59.4260846Z Generating XML reports... 2022-12-01T10:10:59.4261166Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101045.xml 2022-12-01T10:10:59.4261395Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4261666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4261788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4262070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4262209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4262214Z 2022-12-01T10:10:59.4262272Z Running tests... 2022-12-01T10:10:59.4262469Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4262656Z test_reduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-12-01T10:10:59.4262661Z 2022-12-01T10:10:59.4262893Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4262967Z Ran 1 test in 0.002s 2022-12-01T10:10:59.4262972Z 2022-12-01T10:10:59.4263041Z OK (skipped=1) 2022-12-01T10:10:59.4263045Z 2022-12-01T10:10:59.4263128Z Generating XML reports... 2022-12-01T10:10:59.4263447Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101046.xml 2022-12-01T10:10:59.4263703Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4263961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4264081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4264357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4264491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4264495Z 2022-12-01T10:10:59.4264567Z Running tests... 2022-12-01T10:10:59.4264757Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4264961Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4264965Z 2022-12-01T10:10:59.4265157Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4265232Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4265237Z 2022-12-01T10:10:59.4265295Z OK (skipped=1) 2022-12-01T10:10:59.4265299Z 2022-12-01T10:10:59.4265379Z Generating XML reports... 2022-12-01T10:10:59.4265695Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101047.xml 2022-12-01T10:10:59.4265919Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4266189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4266311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4266584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4266711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4266719Z 2022-12-01T10:10:59.4266777Z Running tests... 2022-12-01T10:10:59.4266963Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4267156Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4267161Z 2022-12-01T10:10:59.4267345Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4267418Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4267423Z 2022-12-01T10:10:59.4267497Z OK (skipped=1) 2022-12-01T10:10:59.4267502Z 2022-12-01T10:10:59.4267584Z Generating XML reports... 2022-12-01T10:10:59.4267898Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101049.xml 2022-12-01T10:10:59.4268125Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4268379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4268501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4268777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4268910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4268915Z 2022-12-01T10:10:59.4268984Z Running tests... 2022-12-01T10:10:59.4269171Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4269389Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.004s) 2022-12-01T10:10:59.4269395Z 2022-12-01T10:10:59.4269587Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4269661Z Ran 1 test in 0.004s 2022-12-01T10:10:59.4269666Z 2022-12-01T10:10:59.4269724Z OK (skipped=1) 2022-12-01T10:10:59.4269728Z 2022-12-01T10:10:59.4269810Z Generating XML reports... 2022-12-01T10:10:59.4270158Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101050.xml 2022-12-01T10:10:59.4270384Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4270652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4270771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4271046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4271181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4271186Z 2022-12-01T10:10:59.4271259Z Running tests... 2022-12-01T10:10:59.4271440Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4271628Z test_scatter_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4271635Z 2022-12-01T10:10:59.4271828Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4271904Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4271908Z 2022-12-01T10:10:59.4271978Z OK (skipped=1) 2022-12-01T10:10:59.4271982Z 2022-12-01T10:10:59.4272065Z Generating XML reports... 2022-12-01T10:10:59.4272386Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101051.xml 2022-12-01T10:10:59.4272614Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4272873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4272991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4273265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4273399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4273404Z 2022-12-01T10:10:59.4273475Z Running tests... 2022-12-01T10:10:59.4273668Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4273853Z test_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4273858Z 2022-12-01T10:10:59.4274048Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4274118Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4274123Z 2022-12-01T10:10:59.4274180Z OK (skipped=1) 2022-12-01T10:10:59.4274184Z 2022-12-01T10:10:59.4274264Z Generating XML reports... 2022-12-01T10:10:59.4274584Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101053.xml 2022-12-01T10:10:59.4274811Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4275088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4275208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4275489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4275621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4275654Z 2022-12-01T10:10:59.4275725Z Running tests... 2022-12-01T10:10:59.4275907Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4276094Z test_scatter_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4276099Z 2022-12-01T10:10:59.4276288Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4276359Z Ran 1 test in 0.002s 2022-12-01T10:10:59.4276400Z 2022-12-01T10:10:59.4276471Z OK (skipped=1) 2022-12-01T10:10:59.4276475Z 2022-12-01T10:10:59.4276556Z Generating XML reports... 2022-12-01T10:10:59.4276875Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101054.xml 2022-12-01T10:10:59.4277104Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4277369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4277491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4277767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4277897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4277902Z 2022-12-01T10:10:59.4277969Z Running tests... 2022-12-01T10:10:59.4278165Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4278488Z test_send_recv (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:10:59.4278493Z 2022-12-01T10:10:59.4278682Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4278755Z Ran 1 test in 0.001s 2022-12-01T10:10:59.4278760Z 2022-12-01T10:10:59.4278818Z OK (skipped=1) 2022-12-01T10:10:59.4278834Z 2022-12-01T10:10:59.4278904Z Generating XML reports... 2022-12-01T10:10:59.4279224Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101056.xml 2022-12-01T10:10:59.4279452Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4279724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4279847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4280127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4280260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4280265Z 2022-12-01T10:10:59.4280334Z Running tests... 2022-12-01T10:10:59.4280514Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4280697Z test_common_errors (__main__.RendezvousEnvTest) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-12-01T10:10:59.4280702Z 2022-12-01T10:10:59.4280890Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4280963Z Ran 1 test in 0.004s 2022-12-01T10:10:59.4280967Z 2022-12-01T10:10:59.4281039Z OK (skipped=1) 2022-12-01T10:10:59.4281043Z 2022-12-01T10:10:59.4281124Z Generating XML reports... 2022-12-01T10:10:59.4281434Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221201101057.xml 2022-12-01T10:10:59.4281664Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-12-01T10:10:59.4281934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:10:59.4282043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:10:59.4282368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:10:59.4282504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:10:59.4282509Z 2022-12-01T10:10:59.4282577Z Running tests... 2022-12-01T10:10:59.4282767Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4282951Z test_default_store_timeout_nccl (__main__.TimeoutTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:10:59.4282991Z 2022-12-01T10:10:59.4283183Z ---------------------------------------------------------------------- 2022-12-01T10:10:59.4283257Z Ran 1 test in 0.000s 2022-12-01T10:10:59.4283262Z 2022-12-01T10:10:59.4283320Z OK (skipped=1) 2022-12-01T10:10:59.4283336Z 2022-12-01T10:10:59.4283406Z Generating XML reports... 2022-12-01T10:10:59.4283695Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221201101058.xml 2022-12-01T10:10:59.4283700Z 2022-12-01T10:10:59.4284073Z ##[endgroup] 2022-12-01T10:10:59.4284399Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_290344ik) 2022-12-01T10:10:59.4284404Z 2022-12-01T10:10:59.4284608Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-12-01 10:10:59.360694] 2022-12-01T10:10:59.4284970Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:10:59.361183] 2022-12-01T10:11:10.2337070Z 2022-12-01T10:11:10.2337794Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint 2022-12-01T10:11:10.2339207Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_79mv0frm) 2022-12-01T10:11:10.2340300Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint 2022-12-01T10:11:10.2340683Z 2022-12-01T10:11:10.2340804Z Running tests... 2022-12-01T10:11:10.2341366Z ---------------------------------------------------------------------- 2022-12-01T10:11:10.2342118Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7461 2022-12-01T10:11:10.2342896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7462 2022-12-01T10:11:10.2343501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7463 2022-12-01T10:11:10.2343920Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7464 2022-12-01T10:11:10.2344391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2344727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2345150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2345608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2346213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:10.2347072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2347678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2348226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2348575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2348955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:10.2349430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2350002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2350477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2350830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2351192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:11:10.2351741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2352105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2352533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2352936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2353277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:11:10.2353819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2354396Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2354917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2355469Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2355906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:11:10.2356262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:10.2356662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:11:10.2356997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:10.2357282Z dist init r=1, world=4 2022-12-01T10:11:10.2357485Z dist init r=2, world=4 2022-12-01T10:11:10.2357652Z dist init r=3, world=4 2022-12-01T10:11:10.2357830Z dist init r=0, world=4 2022-12-01T10:11:10.2358079Z skip: Need at least 2 CUDA devices (1.641s) 2022-12-01T10:11:10.2358730Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7517 2022-12-01T10:11:10.2359386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7518 2022-12-01T10:11:10.2359781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7519 2022-12-01T10:11:10.2360093Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7520 2022-12-01T10:11:10.2360619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2360955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2361438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2361791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2362195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:10.2362668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2363051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2363481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2363973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2364298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:11:10.2364843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2365211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2365670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2366119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2366458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:11:10.2367024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2367537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2367899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2368364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2368740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2369061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:10.2369627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2370217Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2370778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2371163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:11:10.2371548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:10.2371909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:11:10.2372256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:10.2372570Z dist init r=3, world=4 2022-12-01T10:11:10.2372756Z dist init r=0, world=4 2022-12-01T10:11:10.2372933Z dist init r=2, world=4 2022-12-01T10:11:10.2373142Z dist init r=1, world=4 2022-12-01T10:11:10.2373365Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:11:10.2373802Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7573 2022-12-01T10:11:10.2374339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7574 2022-12-01T10:11:10.2374654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7575 2022-12-01T10:11:10.2375041Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7576 2022-12-01T10:11:10.2375497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2375820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2376285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2376664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2377059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:10.2377569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2377904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2378334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2378741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2379076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:10.2379605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2379935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2380344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2380687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2381022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:11:10.2381513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2382002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2382396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2382832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2383165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2383500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:11:10.2383997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2384517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2385019Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2385405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:11:10.2385755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:10.2386098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:10.2386428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:11:10.2386686Z dist init r=2, world=4 2022-12-01T10:11:10.2386871Z dist init r=0, world=4 2022-12-01T10:11:10.2387038Z dist init r=1, world=4 2022-12-01T10:11:10.2387218Z dist init r=3, world=4 2022-12-01T10:11:10.2387427Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:11:10.2387847Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7629 2022-12-01T10:11:10.2388316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7630 2022-12-01T10:11:10.2388645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7631 2022-12-01T10:11:10.2388970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7632 2022-12-01T10:11:10.2389475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2389807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2390240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2390614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2390952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:10.2391419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2391744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2392160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2392534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2392872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:10.2393339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2393651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2394074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2394419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2394741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:11:10.2395235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2395731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2396058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2396467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2396816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2397154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:11:10.2397648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2398150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2398791Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2399180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:10.2399517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:10.2399861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:11:10.2400200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:11:10.2400464Z dist init r=0, world=4 2022-12-01T10:11:10.2400633Z dist init r=1, world=4 2022-12-01T10:11:10.2400813Z dist init r=2, world=4 2022-12-01T10:11:10.2400994Z dist init r=3, world=4 2022-12-01T10:11:10.2401187Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:11:10.2401627Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7685 2022-12-01T10:11:10.2402098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7686 2022-12-01T10:11:10.2402427Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7687 2022-12-01T10:11:10.2402809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7688 2022-12-01T10:11:10.2403322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2403657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2404077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2404426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2404765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:10.2405281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2405594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2406024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2406371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2406697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:11:10.2407164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2407495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2407920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2408250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2408583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:10.2409081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2409576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2409891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2410320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2410667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2410988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:11:10.2411485Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2412007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2412523Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2412894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:11:10.2413245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:10.2413583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:10.2413928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:11:10.2414178Z dist init r=2, world=4 2022-12-01T10:11:10.2414366Z dist init r=0, world=4 2022-12-01T10:11:10.2414551Z dist init r=1, world=4 2022-12-01T10:11:10.2414715Z dist init r=3, world=4 2022-12-01T10:11:10.2414920Z skip: Need at least 2 CUDA devices (1.417s) 2022-12-01T10:11:10.2415355Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7741 2022-12-01T10:11:10.2415819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7742 2022-12-01T10:11:10.2416166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7743 2022-12-01T10:11:10.2416491Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7744 2022-12-01T10:11:10.2417001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2417323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2417755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2418138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2418474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:10.2418933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2419264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2419691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2420022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2420354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:10.2420825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2421153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2421564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2421908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2422248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:11:10.2422747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2423231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2423557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2423984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2424318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2424651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:11:10.2425143Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2425663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2426169Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2426555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:11:10.2426902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:10.2427247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:11:10.2427579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:10.2427836Z dist init r=3, world=4 2022-12-01T10:11:10.2428019Z dist init r=2, world=4 2022-12-01T10:11:10.2428185Z dist init r=0, world=4 2022-12-01T10:11:10.2428363Z dist init r=1, world=4 2022-12-01T10:11:10.2428569Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:11:10.2429123Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7797 2022-12-01T10:11:10.2429593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7798 2022-12-01T10:11:10.2429920Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7799 2022-12-01T10:11:10.2430242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7800 2022-12-01T10:11:10.2430731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2431063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2431488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2431821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2432158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:10.2432628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2432957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2433369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2433722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2434058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:11:10.2434527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2434837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2435263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2435604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2435922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:10.2436415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2436911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:10.2437242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:10.2437657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:10.2438001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:10.2438585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:11:10.2439091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2439598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2440120Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:11:10.2440509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:10.2440852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:11:10.2441203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:11:10.2441541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:10.2441802Z dist init r=1, world=4 2022-12-01T10:11:10.2442034Z dist init r=3, world=4 2022-12-01T10:11:10.2442218Z dist init r=0, world=4 2022-12-01T10:11:10.2442398Z dist init r=2, world=4 2022-12-01T10:11:10.2442592Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:11:10.2443538Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71349 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s) 2022-12-01T10:11:10.2444098Z 2022-12-01T10:11:10.2444301Z ---------------------------------------------------------------------- 2022-12-01T10:11:10.2444550Z Ran 8 tests in 9.645s 2022-12-01T10:11:10.2444665Z 2022-12-01T10:11:10.2444738Z OK (skipped=8) 2022-12-01T10:11:10.2444834Z 2022-12-01T10:11:10.2444925Z Generating XML reports... 2022-12-01T10:11:10.2445391Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221201101100.xml 2022-12-01T10:11:10.2445659Z 2022-12-01T10:11:10.2445963Z ##[endgroup] 2022-12-01T10:11:10.2446418Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_79mv0frm) 2022-12-01T10:11:10.2446690Z 2022-12-01T10:11:10.2446887Z Running distributed/fsdp/test_fsdp_misc ... [2022-12-01 10:11:10.234295] 2022-12-01T10:11:10.2447401Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:11:10.234718] 2022-12-01T10:11:31.0237771Z 2022-12-01T10:11:31.0238627Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_misc 2022-12-01T10:11:31.0239757Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_u3w9mu2a) 2022-12-01T10:11:31.0240614Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc 2022-12-01T10:11:31.0240965Z 2022-12-01T10:11:31.0241059Z Running tests... 2022-12-01T10:11:31.0241519Z ---------------------------------------------------------------------- 2022-12-01T10:11:31.0241962Z test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0242620Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7864 2022-12-01T10:11:31.0243259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7865 2022-12-01T10:11:31.0244042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0244566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0245315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0245906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0246521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0247375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0247968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0248810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0249425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0249962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0251081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0252082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0252793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0253427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0254012Z dist init r=0, world=2 2022-12-01T10:11:31.0254327Z dist init r=1, world=2 2022-12-01T10:11:31.0254683Z skip: Need at least 2 CUDA devices (1.432s) 2022-12-01T10:11:31.0255095Z test_device_id_auto_wrap (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0255747Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7892 2022-12-01T10:11:31.0256381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7893 2022-12-01T10:11:31.0257307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0257909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0258708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0259358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0259928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0260821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0261344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0262005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0262560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0263087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0263878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0264711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0265369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0265950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0266364Z dist init r=0, world=2 2022-12-01T10:11:31.0266627Z dist init r=1, world=2 2022-12-01T10:11:31.0266959Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:11:31.0267352Z test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0268149Z Tests that passing a CPU module to FSDP preserves that the wrapped ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7920 2022-12-01T10:11:31.0268848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7921 2022-12-01T10:11:31.0269621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0270164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0270891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0271526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0272139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0272997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0273721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0274494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0275128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0275704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0276652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0277506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0278168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0278921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0279340Z dist init r=1, world=2 2022-12-01T10:11:31.0279624Z dist init r=0, world=2 2022-12-01T10:11:31.0279942Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:11:31.0280362Z test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0280913Z Ensures that even if device_id is specified but we have ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7948 2022-12-01T10:11:31.0281500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7949 2022-12-01T10:11:31.0282236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0282741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0283445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0283989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0284527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0285316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0285892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0286646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0287248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0287765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0288423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0288959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0289346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0289691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0289945Z dist init r=1, world=2 2022-12-01T10:11:31.0290126Z dist init r=0, world=2 2022-12-01T10:11:31.0290334Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:11:31.0290581Z test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0290930Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7976 2022-12-01T10:11:31.0291299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7977 2022-12-01T10:11:31.0291754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0292172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0292607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0292956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0293280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0293749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0294121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0294547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0294877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0295211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0295709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0296230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0296601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0297030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0297291Z dist init r=1, world=2 2022-12-01T10:11:31.0297457Z dist init r=0, world=2 2022-12-01T10:11:31.0297665Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:11:31.0297919Z test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0298250Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8004 2022-12-01T10:11:31.0298617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8005 2022-12-01T10:11:31.0299070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0299397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0299812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0300159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0300498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0300963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0301277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0301702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0302048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0302369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0302859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0303376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0303765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0304098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0304351Z dist init r=1, world=2 2022-12-01T10:11:31.0304528Z dist init r=0, world=2 2022-12-01T10:11:31.0304724Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:11:31.0305155Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8032 2022-12-01T10:11:31.0305585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8033 2022-12-01T10:11:31.0306035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0306351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0306809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0307151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0307485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0307940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0308265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0308691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0309020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0309350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0309846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0310363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0310733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0311077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0311336Z dist init r=0, world=2 2022-12-01T10:11:31.0311503Z dist init r=1, world=2 2022-12-01T10:11:31.0311711Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:11:31.0312131Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8060 2022-12-01T10:11:31.0312581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8061 2022-12-01T10:11:31.0313020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0313346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0313771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0314113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0314435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0314901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0315223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0315633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0315976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0316309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0316798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0317302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0317713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0318060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0318509Z dist init r=1, world=2 2022-12-01T10:11:31.0318676Z dist init r=0, world=2 2022-12-01T10:11:31.0318883Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:11:31.0319280Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8088 2022-12-01T10:11:31.0319755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8089 2022-12-01T10:11:31.0320214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0320543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0320974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0321303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0321640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0322104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0322419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0322843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0323188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0323520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0324002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0324519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0324905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0325252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0325498Z dist init r=1, world=2 2022-12-01T10:11:31.0325677Z dist init r=0, world=2 2022-12-01T10:11:31.0325886Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:11:31.0326288Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8116 2022-12-01T10:11:31.0326736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8117 2022-12-01T10:11:31.0327190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0327521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0327935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0328279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0328615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0329069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0329396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0329818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0330201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0330522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0331016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0331534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0331951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0332283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0332544Z dist init r=1, world=2 2022-12-01T10:11:31.0332722Z dist init r=0, world=2 2022-12-01T10:11:31.0332918Z skip: Need at least 2 CUDA devices (1.211s) 2022-12-01T10:11:31.0333255Z test_fsdp_namedtuple (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8144 2022-12-01T10:11:31.0333629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8145 2022-12-01T10:11:31.0334069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0334398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0334826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0335172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0335492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0335959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0336285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0336777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0337105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0337441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0337937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0338448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0338832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0339177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0339435Z dist init r=1, world=2 2022-12-01T10:11:31.0339602Z dist init r=0, world=2 2022-12-01T10:11:31.0339810Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:11:31.0340164Z test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8172 2022-12-01T10:11:31.0340537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8173 2022-12-01T10:11:31.0340991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0341320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0341744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0342073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0342405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0342912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0343227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0343656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0343997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0344329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0344860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0345379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0345765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0346113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0346359Z dist init r=0, world=2 2022-12-01T10:11:31.0346538Z dist init r=1, world=2 2022-12-01T10:11:31.0346744Z skip: Need at least 2 CUDA devices (1.212s) 2022-12-01T10:11:31.0346986Z test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0347363Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8200 2022-12-01T10:11:31.0347755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8201 2022-12-01T10:11:31.0348203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0348518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0348945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0349290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0349613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0350074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0350401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0350830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0351161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0351492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0351981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0352488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0352875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0353221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0353481Z dist init r=0, world=2 2022-12-01T10:11:31.0353648Z dist init r=1, world=2 2022-12-01T10:11:31.0353856Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:11:31.0354118Z test_module_device_mismatches_device_id (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0354489Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8228 2022-12-01T10:11:31.0354880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8229 2022-12-01T10:11:31.0355363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0355694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0356113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0356459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0356790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0357293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0357606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0358031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0358489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0358812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0359305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0359825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0360206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0360539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0360790Z dist init r=1, world=2 2022-12-01T10:11:31.0360968Z dist init r=0, world=2 2022-12-01T10:11:31.0361163Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:11:31.0361417Z test_multi_device_not_supported (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0361901Z Tests that wrapping a multi-device module (i.e. with submodules on ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8256 2022-12-01T10:11:31.0362297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8257 2022-12-01T10:11:31.0362733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0363062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0363492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0363826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0364160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0364623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0364947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0365359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0365700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0366031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0366508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0367031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0367414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0367758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0368005Z dist init r=1, world=2 2022-12-01T10:11:31.0368184Z dist init r=0, world=2 2022-12-01T10:11:31.0368441Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:11:31.0368661Z test_no_params (__main__.TestFSDPMisc) 2022-12-01T10:11:31.0369007Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8284 2022-12-01T10:11:31.0369393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8285 2022-12-01T10:11:31.0369842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0370195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0370620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0370962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0371301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:11:31.0371753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:11:31.0372076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:11:31.0372499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:11:31.0372826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:11:31.0373158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:11:31.0373657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0374177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:11:31.0374551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:11:31.0374896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:11:31.0375153Z dist init r=1, world=2 2022-12-01T10:11:31.0375318Z dist init r=0, world=2 2022-12-01T10:11:31.0375522Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:11:31.0375658Z 2022-12-01T10:11:31.0375859Z ---------------------------------------------------------------------- 2022-12-01T10:11:31.0376106Z Ran 16 tests in 19.593s 2022-12-01T10:11:31.0376208Z 2022-12-01T10:11:31.0376279Z OK (skipped=16) 2022-12-01T10:11:31.0376387Z 2022-12-01T10:11:31.0376472Z Generating XML reports... 2022-12-01T10:11:31.0376965Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221201101111.xml 2022-12-01T10:11:31.0377213Z 2022-12-01T10:11:31.0377541Z ##[endgroup] 2022-12-01T10:11:31.0377989Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_u3w9mu2a) 2022-12-01T10:11:31.0378246Z 2022-12-01T10:11:31.0378422Z Running distributed/test_store ... [2022-12-01 10:11:31.024376] 2022-12-01T10:11:31.0378921Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:11:31.024724] 2022-12-01T10:12:20.0647223Z 2022-12-01T10:12:20.0648877Z Expand the folded group to see the log file of distributed/test_store 2022-12-01T10:12:20.0649916Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_y835s00g) 2022-12-01T10:12:20.0650616Z , <__main__.FileStoreTest testMethod=test_set_get>]> 2022-12-01T10:12:20.0651127Z test_compare_set (__main__.FileStoreTest) 2022-12-01T10:12:20.0684090Z test_set_get (__main__.FileStoreTest) 2022-12-01T10:12:20.0685037Z , <__main__.HashStoreTest testMethod=test_set_get>]> 2022-12-01T10:12:20.0685671Z test_compare_set (__main__.HashStoreTest) 2022-12-01T10:12:20.0686070Z test_set_get (__main__.HashStoreTest) 2022-12-01T10:12:20.0686636Z , <__main__.PrefixFileStoreTest testMethod=test_set_get>]> 2022-12-01T10:12:20.0687067Z test_compare_set (__main__.PrefixFileStoreTest) 2022-12-01T10:12:20.0687409Z test_set_get (__main__.PrefixFileStoreTest) 2022-12-01T10:12:20.0687713Z ]> 2022-12-01T10:12:20.0688033Z test_get_underlying_store (__main__.PrefixStoreTest) 2022-12-01T10:12:20.0688423Z , <__main__.PrefixTCPStoreTest testMethod=test_set_get>]> 2022-12-01T10:12:20.0689246Z test_compare_set (__main__.PrefixTCPStoreTest) 2022-12-01T10:12:20.0689599Z test_set_get (__main__.PrefixTCPStoreTest) 2022-12-01T10:12:20.0690116Z ]> 2022-12-01T10:12:20.0690613Z test_set_get (__main__.PythonStoreTest) 2022-12-01T10:12:20.0691095Z ]> 2022-12-01T10:12:20.0691389Z test_nominal (__main__.RendezvousEnvTest) 2022-12-01T10:12:20.0691818Z , <__main__.RendezvousFileTest testMethod=test_nominal>]> 2022-12-01T10:12:20.0692502Z test_common_errors (__main__.RendezvousFileTest) 2022-12-01T10:12:20.0692898Z test_nominal (__main__.RendezvousFileTest) 2022-12-01T10:12:20.0693617Z , <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]> 2022-12-01T10:12:20.0694506Z test_common_errors (__main__.RendezvousTCPTest) 2022-12-01T10:12:20.0694820Z test_dns_timeout (__main__.RendezvousTCPTest) 2022-12-01T10:12:20.0695045Z test_nominal (__main__.RendezvousTCPTest) 2022-12-01T10:12:20.0695295Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) 2022-12-01T10:12:20.0695721Z , <__main__.RendezvousTest testMethod=test_url_with_node_params>]> 2022-12-01T10:12:20.0696190Z test_unknown_handler (__main__.RendezvousTest) 2022-12-01T10:12:20.0696629Z test_url_with_node_params (__main__.RendezvousTest) 2022-12-01T10:12:20.0697928Z , <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]> 2022-12-01T10:12:20.0698637Z test_address_already_in_use (__main__.TCPStoreTest) 2022-12-01T10:12:20.0698958Z test_compare_set (__main__.TCPStoreTest) 2022-12-01T10:12:20.0699453Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) 2022-12-01T10:12:20.0699926Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) 2022-12-01T10:12:20.0700330Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) 2022-12-01T10:12:20.0700746Z test_multitenancy (__main__.TCPStoreTest) 2022-12-01T10:12:20.0701114Z test_numkeys_delkeys (__main__.TCPStoreTest) 2022-12-01T10:12:20.0701490Z test_set_get (__main__.TCPStoreTest) 2022-12-01T10:12:20.0702215Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0703221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0703829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0704358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0704708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0704918Z 2022-12-01T10:12:20.0704978Z Running tests... 2022-12-01T10:12:20.0705284Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0705615Z test_compare_set (__main__.FileStoreTest) ... ok (0.223s) 2022-12-01T10:12:20.0705765Z 2022-12-01T10:12:20.0705963Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0706204Z Ran 1 test in 0.223s 2022-12-01T10:12:20.0706303Z 2022-12-01T10:12:20.0706364Z OK 2022-12-01T10:12:20.0706458Z 2022-12-01T10:12:20.0706545Z Generating XML reports... 2022-12-01T10:12:20.0706971Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221201101132.xml 2022-12-01T10:12:20.0707416Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0707875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0708209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0708636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0709030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0709194Z 2022-12-01T10:12:20.0709267Z Running tests... 2022-12-01T10:12:20.0709571Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0709839Z test_set_get (__main__.FileStoreTest) ... ok (0.219s) 2022-12-01T10:12:20.0709981Z 2022-12-01T10:12:20.0710175Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0710408Z Ran 1 test in 0.220s 2022-12-01T10:12:20.0710518Z 2022-12-01T10:12:20.0710578Z OK 2022-12-01T10:12:20.0710655Z 2022-12-01T10:12:20.0710738Z Generating XML reports... 2022-12-01T10:12:20.0711150Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221201101134.xml 2022-12-01T10:12:20.0711604Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0712047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0712375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0712803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0713148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0713300Z 2022-12-01T10:12:20.0713373Z Running tests... 2022-12-01T10:12:20.0713670Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0713954Z test_compare_set (__main__.HashStoreTest) ... ok (0.222s) 2022-12-01T10:12:20.0714103Z 2022-12-01T10:12:20.0714282Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0714521Z Ran 1 test in 0.222s 2022-12-01T10:12:20.0714631Z 2022-12-01T10:12:20.0714690Z OK 2022-12-01T10:12:20.0714781Z 2022-12-01T10:12:20.0714864Z Generating XML reports... 2022-12-01T10:12:20.0715259Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221201101135.xml 2022-12-01T10:12:20.0715705Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0716209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0716529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0716958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0717300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0717465Z 2022-12-01T10:12:20.0717538Z Running tests... 2022-12-01T10:12:20.0717853Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0718133Z test_set_get (__main__.HashStoreTest) ... ok (0.218s) 2022-12-01T10:12:20.0718503Z 2022-12-01T10:12:20.0718815Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0719243Z Ran 1 test in 0.218s 2022-12-01T10:12:20.0719446Z 2022-12-01T10:12:20.0719547Z OK 2022-12-01T10:12:20.0719665Z 2022-12-01T10:12:20.0719752Z Generating XML reports... 2022-12-01T10:12:20.0720176Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221201101136.xml 2022-12-01T10:12:20.0720617Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0721074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0721409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0721827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0722176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0722341Z 2022-12-01T10:12:20.0722412Z Running tests... 2022-12-01T10:12:20.0722718Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0723002Z test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.221s) 2022-12-01T10:12:20.0723163Z 2022-12-01T10:12:20.0723360Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0723594Z Ran 1 test in 0.221s 2022-12-01T10:12:20.0723703Z 2022-12-01T10:12:20.0723750Z OK 2022-12-01T10:12:20.0723840Z 2022-12-01T10:12:20.0723925Z Generating XML reports... 2022-12-01T10:12:20.0724364Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221201101138.xml 2022-12-01T10:12:20.0724830Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0725273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0725603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0726034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0726367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0726533Z 2022-12-01T10:12:20.0726605Z Running tests... 2022-12-01T10:12:20.0726908Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0727202Z test_set_get (__main__.PrefixFileStoreTest) ... ok (0.222s) 2022-12-01T10:12:20.0727341Z 2022-12-01T10:12:20.0727534Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0727773Z Ran 1 test in 0.222s 2022-12-01T10:12:20.0727882Z 2022-12-01T10:12:20.0727946Z OK 2022-12-01T10:12:20.0728040Z 2022-12-01T10:12:20.0728110Z Generating XML reports... 2022-12-01T10:12:20.0728541Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221201101139.xml 2022-12-01T10:12:20.0729010Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0729466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0729882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0730316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0730667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0730832Z 2022-12-01T10:12:20.0758826Z Running tests... 2022-12-01T10:12:20.0759363Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0760222Z test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.002s) 2022-12-01T10:12:20.0764506Z 2022-12-01T10:12:20.0764737Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0764971Z Ran 1 test in 0.002s 2022-12-01T10:12:20.0778315Z 2022-12-01T10:12:20.0778391Z OK 2022-12-01T10:12:20.0778493Z 2022-12-01T10:12:20.0778586Z Generating XML reports... 2022-12-01T10:12:20.0779074Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221201101140.xml 2022-12-01T10:12:20.0779537Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0780003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0780339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0780759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0781113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0781277Z 2022-12-01T10:12:20.0781351Z Running tests... 2022-12-01T10:12:20.0781655Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0781940Z test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.221s) 2022-12-01T10:12:20.0782100Z 2022-12-01T10:12:20.0782293Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0782530Z Ran 1 test in 0.222s 2022-12-01T10:12:20.0782642Z 2022-12-01T10:12:20.0782689Z OK 2022-12-01T10:12:20.0782780Z 2022-12-01T10:12:20.0782863Z Generating XML reports... 2022-12-01T10:12:20.0783292Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221201101141.xml 2022-12-01T10:12:20.0783765Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0784212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0784640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0785073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0785418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0785568Z 2022-12-01T10:12:20.0785640Z Running tests... 2022-12-01T10:12:20.0785946Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0786236Z test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.223s) 2022-12-01T10:12:20.0786388Z 2022-12-01T10:12:20.0786569Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0786804Z Ran 1 test in 0.223s 2022-12-01T10:12:20.0786916Z 2022-12-01T10:12:20.0786977Z OK 2022-12-01T10:12:20.0787066Z 2022-12-01T10:12:20.0787142Z Generating XML reports... 2022-12-01T10:12:20.0787573Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221201101143.xml 2022-12-01T10:12:20.0788045Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0788505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0788825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0789366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0789718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0789883Z 2022-12-01T10:12:20.0789958Z Running tests... 2022-12-01T10:12:20.0790248Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0790534Z test_set_get (__main__.PythonStoreTest) ... ok (0.220s) 2022-12-01T10:12:20.0790718Z 2022-12-01T10:12:20.0790913Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0791136Z Ran 1 test in 0.220s 2022-12-01T10:12:20.0791248Z 2022-12-01T10:12:20.0791307Z OK 2022-12-01T10:12:20.0791398Z 2022-12-01T10:12:20.0791483Z Generating XML reports... 2022-12-01T10:12:20.0791890Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221201101144.xml 2022-12-01T10:12:20.0792354Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0792813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0793144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0793560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0793914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0794076Z 2022-12-01T10:12:20.0794149Z Running tests... 2022-12-01T10:12:20.0794450Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0794724Z test_nominal (__main__.RendezvousEnvTest) ... ok (0.220s) 2022-12-01T10:12:20.0794875Z 2022-12-01T10:12:20.0795069Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0795306Z Ran 1 test in 0.220s 2022-12-01T10:12:20.0795416Z 2022-12-01T10:12:20.0795466Z OK 2022-12-01T10:12:20.0795555Z 2022-12-01T10:12:20.0795643Z Generating XML reports... 2022-12-01T10:12:20.0796072Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221201101145.xml 2022-12-01T10:12:20.0796537Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0796982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0797315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0797740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0798074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0798472Z 2022-12-01T10:12:20.0798560Z Running tests... 2022-12-01T10:12:20.0798874Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0799174Z test_common_errors (__main__.RendezvousFileTest) ... ok (0.220s) 2022-12-01T10:12:20.0799321Z 2022-12-01T10:12:20.0799517Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0799759Z Ran 1 test in 0.221s 2022-12-01T10:12:20.0799870Z 2022-12-01T10:12:20.0799932Z OK 2022-12-01T10:12:20.0800024Z 2022-12-01T10:12:20.0800095Z Generating XML reports... 2022-12-01T10:12:20.0800529Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221201101147.xml 2022-12-01T10:12:20.0800997Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0801454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0801772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0802265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0802616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0802782Z 2022-12-01T10:12:20.0802840Z Running tests... 2022-12-01T10:12:20.0803142Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0803435Z test_nominal (__main__.RendezvousFileTest) ... ok (0.221s) 2022-12-01T10:12:20.0803587Z 2022-12-01T10:12:20.0803821Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0804041Z Ran 1 test in 0.222s 2022-12-01T10:12:20.0804152Z 2022-12-01T10:12:20.0804215Z OK 2022-12-01T10:12:20.0804306Z 2022-12-01T10:12:20.0804390Z Generating XML reports... 2022-12-01T10:12:20.0804806Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221201101148.xml 2022-12-01T10:12:20.0805271Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0805730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0806060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0806477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0806825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0806993Z 2022-12-01T10:12:20.0807066Z Running tests... 2022-12-01T10:12:20.0807355Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0807649Z test_common_errors (__main__.RendezvousTCPTest) ... ok (0.221s) 2022-12-01T10:12:20.0807807Z 2022-12-01T10:12:20.0808000Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0808241Z Ran 1 test in 0.221s 2022-12-01T10:12:20.0808338Z 2022-12-01T10:12:20.0808399Z OK 2022-12-01T10:12:20.0808489Z 2022-12-01T10:12:20.0808576Z Generating XML reports... 2022-12-01T10:12:20.0809002Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201101149.xml 2022-12-01T10:12:20.0809456Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0809910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0810237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0810671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0811000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0811164Z 2022-12-01T10:12:20.0811238Z Running tests... 2022-12-01T10:12:20.0811541Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0812086Z test_dns_timeout (__main__.RendezvousTCPTest) ... [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known). 2022-12-01T10:12:20.0812526Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456). 2022-12-01T10:12:20.0812787Z ok (0.257s) 2022-12-01T10:12:20.0812888Z 2022-12-01T10:12:20.0813086Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0813311Z Ran 1 test in 0.257s 2022-12-01T10:12:20.0813421Z 2022-12-01T10:12:20.0813482Z OK 2022-12-01T10:12:20.0813573Z 2022-12-01T10:12:20.0813655Z Generating XML reports... 2022-12-01T10:12:20.0814067Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201101151.xml 2022-12-01T10:12:20.0814526Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0815018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0815351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0815769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0816112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0816274Z 2022-12-01T10:12:20.0816346Z Running tests... 2022-12-01T10:12:20.0816738Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0817028Z test_nominal (__main__.RendezvousTCPTest) ... ok (0.219s) 2022-12-01T10:12:20.0817177Z 2022-12-01T10:12:20.0817375Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0817616Z Ran 1 test in 0.219s 2022-12-01T10:12:20.0817728Z 2022-12-01T10:12:20.0817777Z OK 2022-12-01T10:12:20.0817866Z 2022-12-01T10:12:20.0817951Z Generating XML reports... 2022-12-01T10:12:20.0818384Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201101152.xml 2022-12-01T10:12:20.0818837Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0819292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0819624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0820055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0820390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0820553Z 2022-12-01T10:12:20.0820625Z Running tests... 2022-12-01T10:12:20.0820925Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0821216Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... ok (10.262s) 2022-12-01T10:12:20.0821383Z 2022-12-01T10:12:20.0821581Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0821822Z Ran 1 test in 10.262s 2022-12-01T10:12:20.0821933Z 2022-12-01T10:12:20.0821991Z OK 2022-12-01T10:12:20.0822083Z 2022-12-01T10:12:20.0822154Z Generating XML reports... 2022-12-01T10:12:20.0822579Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201101153.xml 2022-12-01T10:12:20.0823038Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0823497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0823814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0824243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0824589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0824754Z 2022-12-01T10:12:20.0824814Z Running tests... 2022-12-01T10:12:20.0825113Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0825398Z test_unknown_handler (__main__.RendezvousTest) ... ok (0.218s) 2022-12-01T10:12:20.0825551Z 2022-12-01T10:12:20.0825745Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0825971Z Ran 1 test in 0.218s 2022-12-01T10:12:20.0826085Z 2022-12-01T10:12:20.0826146Z OK 2022-12-01T10:12:20.0826235Z 2022-12-01T10:12:20.0826319Z Generating XML reports... 2022-12-01T10:12:20.0826723Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221201101205.xml 2022-12-01T10:12:20.0827177Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0827632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0827992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0828411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0828754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0828919Z 2022-12-01T10:12:20.0828992Z Running tests... 2022-12-01T10:12:20.0829279Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0829605Z test_url_with_node_params (__main__.RendezvousTest) ... ok (0.220s) 2022-12-01T10:12:20.0829763Z 2022-12-01T10:12:20.0829959Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0830194Z Ran 1 test in 0.221s 2022-12-01T10:12:20.0830293Z 2022-12-01T10:12:20.0830352Z OK 2022-12-01T10:12:20.0830442Z 2022-12-01T10:12:20.0830526Z Generating XML reports... 2022-12-01T10:12:20.0830939Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221201101206.xml 2022-12-01T10:12:20.0831383Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0831838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0832165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0832592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0832926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0833087Z 2022-12-01T10:12:20.0833160Z Running tests... 2022-12-01T10:12:20.0833461Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0833986Z test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:41875 (errno: 98 - Address already in use). 2022-12-01T10:12:20.0834491Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:41875 (errno: 98 - Address already in use). 2022-12-01T10:12:20.0834835Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-12-01T10:12:20.0835075Z ok (0.224s) 2022-12-01T10:12:20.0835176Z 2022-12-01T10:12:20.0835361Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0835602Z Ran 1 test in 0.224s 2022-12-01T10:12:20.0835715Z 2022-12-01T10:12:20.0835775Z OK 2022-12-01T10:12:20.0835865Z 2022-12-01T10:12:20.0835935Z Generating XML reports... 2022-12-01T10:12:20.0836341Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101207.xml 2022-12-01T10:12:20.0836790Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0837245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0837566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0837994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0838470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0838634Z 2022-12-01T10:12:20.0838707Z Running tests... 2022-12-01T10:12:20.0838998Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0839287Z test_compare_set (__main__.TCPStoreTest) ... ok (0.222s) 2022-12-01T10:12:20.0839432Z 2022-12-01T10:12:20.0839627Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0839850Z Ran 1 test in 0.222s 2022-12-01T10:12:20.0839961Z 2022-12-01T10:12:20.0840022Z OK 2022-12-01T10:12:20.0840113Z 2022-12-01T10:12:20.0840197Z Generating XML reports... 2022-12-01T10:12:20.0840654Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101209.xml 2022-12-01T10:12:20.0841110Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0841567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0841898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0842312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0842698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0842861Z 2022-12-01T10:12:20.0842933Z Running tests... 2022-12-01T10:12:20.0843233Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0843627Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:20.0844188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:12:20.0844484Z ok (0.314s) 2022-12-01T10:12:20.0844584Z 2022-12-01T10:12:20.0844766Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0845002Z Ran 1 test in 0.314s 2022-12-01T10:12:20.0845113Z 2022-12-01T10:12:20.0845173Z OK 2022-12-01T10:12:20.0845263Z 2022-12-01T10:12:20.0845349Z Generating XML reports... 2022-12-01T10:12:20.0845746Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101210.xml 2022-12-01T10:12:20.0846197Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0846653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0846972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0847401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0847747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0847911Z 2022-12-01T10:12:20.0847983Z Running tests... 2022-12-01T10:12:20.0848269Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0848575Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.223s) 2022-12-01T10:12:20.0848746Z 2022-12-01T10:12:20.0848937Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0849160Z Ran 1 test in 0.223s 2022-12-01T10:12:20.0849272Z 2022-12-01T10:12:20.0849332Z OK 2022-12-01T10:12:20.0849421Z 2022-12-01T10:12:20.0849505Z Generating XML reports... 2022-12-01T10:12:20.0849913Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101212.xml 2022-12-01T10:12:20.0850354Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0850807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0851139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0851550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0851898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0852063Z 2022-12-01T10:12:20.0852134Z Running tests... 2022-12-01T10:12:20.0852433Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0852731Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.222s) 2022-12-01T10:12:20.0852905Z 2022-12-01T10:12:20.0853099Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0853363Z Ran 1 test in 0.222s 2022-12-01T10:12:20.0853476Z 2022-12-01T10:12:20.0853523Z OK 2022-12-01T10:12:20.0853613Z 2022-12-01T10:12:20.0853698Z Generating XML reports... 2022-12-01T10:12:20.0854111Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101213.xml 2022-12-01T10:12:20.0854560Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0855001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0855358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0855790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0856124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0856288Z 2022-12-01T10:12:20.0856360Z Running tests... 2022-12-01T10:12:20.0856727Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0857014Z test_multitenancy (__main__.TCPStoreTest) ... ok (0.222s) 2022-12-01T10:12:20.0857163Z 2022-12-01T10:12:20.0857345Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0857583Z Ran 1 test in 0.222s 2022-12-01T10:12:20.0857695Z 2022-12-01T10:12:20.0857756Z OK 2022-12-01T10:12:20.0857845Z 2022-12-01T10:12:20.0857915Z Generating XML reports... 2022-12-01T10:12:20.0858327Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101214.xml 2022-12-01T10:12:20.0858781Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0859238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0859551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0859982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0860327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0860489Z 2022-12-01T10:12:20.0860563Z Running tests... 2022-12-01T10:12:20.0860849Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0861133Z test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.248s) 2022-12-01T10:12:20.0861286Z 2022-12-01T10:12:20.0861478Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0861701Z Ran 1 test in 2.249s 2022-12-01T10:12:20.0861809Z 2022-12-01T10:12:20.0861872Z OK 2022-12-01T10:12:20.0861961Z 2022-12-01T10:12:20.0862043Z Generating XML reports... 2022-12-01T10:12:20.0862439Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101216.xml 2022-12-01T10:12:20.0862894Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-12-01T10:12:20.0863350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:20.0863676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:20.0864086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:20.0864429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:20.0864593Z 2022-12-01T10:12:20.0864665Z Running tests... 2022-12-01T10:12:20.0864952Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0865227Z test_set_get (__main__.TCPStoreTest) ... ok (0.222s) 2022-12-01T10:12:20.0865370Z 2022-12-01T10:12:20.0865563Z ---------------------------------------------------------------------- 2022-12-01T10:12:20.0865798Z Ran 1 test in 0.222s 2022-12-01T10:12:20.0865896Z 2022-12-01T10:12:20.0865956Z OK 2022-12-01T10:12:20.0866045Z 2022-12-01T10:12:20.0866164Z Generating XML reports... 2022-12-01T10:12:20.0866575Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101219.xml 2022-12-01T10:12:20.0866804Z 2022-12-01T10:12:20.0867162Z ##[endgroup] 2022-12-01T10:12:20.0867578Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_y835s00g) 2022-12-01T10:12:20.0867810Z 2022-12-01T10:12:20.0868075Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-12-01 10:12:20.065434] 2022-12-01T10:12:20.0868622Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:12:20.065939] 2022-12-01T10:12:32.4914483Z 2022-12-01T10:12:32.4915405Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_freezing_weights 2022-12-01T10:12:32.4918105Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_f79ssb64) 2022-12-01T10:12:32.4919164Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights 2022-12-01T10:12:32.4919608Z 2022-12-01T10:12:32.4919868Z Running tests... 2022-12-01T10:12:32.4920525Z ---------------------------------------------------------------------- 2022-12-01T10:12:32.4921575Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8684 2022-12-01T10:12:32.4922291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8685 2022-12-01T10:12:32.4922660Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8686 2022-12-01T10:12:32.4922970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8687 2022-12-01T10:12:32.4923452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4923795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4924219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4924572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4925036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:32.4925612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4925934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4926438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4926792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4927135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:32.4927591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4927919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4928342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4928677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4929012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:32.4929487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4929976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4930392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4930739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4931079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:32.4931568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4932159Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4932682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4933205Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4933579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:32.4933933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:32.4934276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:32.4934628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:32.4934879Z dist init r=0, world=4 2022-12-01T10:12:32.4935063Z dist init r=2, world=4 2022-12-01T10:12:32.4935243Z dist init r=1, world=4 2022-12-01T10:12:32.4935408Z dist init r=3, world=4 2022-12-01T10:12:32.4935615Z skip: Need at least 2 CUDA devices (1.638s) 2022-12-01T10:12:32.4936072Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8740 2022-12-01T10:12:32.4936661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8741 2022-12-01T10:12:32.4936981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8742 2022-12-01T10:12:32.4937303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8743 2022-12-01T10:12:32.4937766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4938091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4938521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4938869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4939208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:32.4939664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4939993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4940414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4940745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4941082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:32.4941564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4941895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4942326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4942861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4943377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:32.4944224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4945052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4945634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4946522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4947023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4947362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:32.4947865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4948398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4948903Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4949285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:32.4949635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:32.4949987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:32.4950318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:32.4950575Z dist init r=0, world=4 2022-12-01T10:12:32.4950758Z dist init r=1, world=4 2022-12-01T10:12:32.4950922Z dist init r=3, world=4 2022-12-01T10:12:32.4951102Z dist init r=2, world=4 2022-12-01T10:12:32.4951310Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:12:32.4951762Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8796 2022-12-01T10:12:32.4952254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8797 2022-12-01T10:12:32.4952581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8798 2022-12-01T10:12:32.4952906Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8799 2022-12-01T10:12:32.4953352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4953688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4954120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4954475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4954801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:32.4955270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4955697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4956134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4956475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4956799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:32.4957269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4957643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4958074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4958634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4958979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:32.4959483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4960072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4960390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4960822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4961177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4961498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:32.4961996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4962518Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4963039Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4963407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:32.4963757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:32.4964101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:32.4964446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:32.4964691Z dist init r=0, world=4 2022-12-01T10:12:32.4964874Z dist init r=1, world=4 2022-12-01T10:12:32.4965053Z dist init r=2, world=4 2022-12-01T10:12:32.4965217Z dist init r=3, world=4 2022-12-01T10:12:32.4965424Z skip: Need at least 2 CUDA devices (1.416s) 2022-12-01T10:12:32.4965883Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8852 2022-12-01T10:12:32.4966359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8853 2022-12-01T10:12:32.4966688Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8854 2022-12-01T10:12:32.4967009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8855 2022-12-01T10:12:32.4967472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4967795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4968227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4968574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4968897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:32.4969369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4969697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4970126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4970499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4970843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:32.4971313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4971641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4972055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4972436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4972772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:32.4973256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4973754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4974083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4974510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4974838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4975173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:32.4975667Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4976190Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4976763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4977151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:32.4977501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:32.4977835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:32.4978182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:32.4978440Z dist init r=2, world=4 2022-12-01T10:12:32.4978627Z dist init r=0, world=4 2022-12-01T10:12:32.4978795Z dist init r=1, world=4 2022-12-01T10:12:32.4978975Z dist init r=3, world=4 2022-12-01T10:12:32.4979180Z skip: Need at least 2 CUDA devices (1.416s) 2022-12-01T10:12:32.4979619Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8908 2022-12-01T10:12:32.4980108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8909 2022-12-01T10:12:32.4980439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8910 2022-12-01T10:12:32.4980766Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8911 2022-12-01T10:12:32.4981203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4981539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4981971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4982306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4982642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:32.4983112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4983479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4983892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4984238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4984574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:32.4985079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4985394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4985821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4986168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4986490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:32.4986954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4987282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4987708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4988038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4988370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:32.4988869Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4989380Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4989901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4990419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.4990807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:32.4991138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:32.4991486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:32.4991827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:32.4992083Z dist init r=0, world=4 2022-12-01T10:12:32.4992251Z dist init r=3, world=4 2022-12-01T10:12:32.4992429Z dist init r=2, world=4 2022-12-01T10:12:32.4992609Z dist init r=1, world=4 2022-12-01T10:12:32.4992801Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:12:32.4993256Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8964 2022-12-01T10:12:32.4993739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8965 2022-12-01T10:12:32.4994067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8966 2022-12-01T10:12:32.4994379Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8967 2022-12-01T10:12:32.4994830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4995161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4995575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4995952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4996288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:32.4996759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4997072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4997501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4997876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.4998329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:32.4998846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.4999184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.4999613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.4999941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5000279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:32.5000746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5001076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5001487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5001827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5002159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:32.5002643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5003168Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5003682Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5004203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5004571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:32.5004926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:32.5005275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:32.5005622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:32.5005871Z dist init r=3, world=4 2022-12-01T10:12:32.5006051Z dist init r=2, world=4 2022-12-01T10:12:32.5006231Z dist init r=1, world=4 2022-12-01T10:12:32.5006394Z dist init r=0, world=4 2022-12-01T10:12:32.5006598Z skip: Need at least 2 CUDA devices (1.416s) 2022-12-01T10:12:32.5007054Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9020 2022-12-01T10:12:32.5007532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9021 2022-12-01T10:12:32.5007863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9022 2022-12-01T10:12:32.5008185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9023 2022-12-01T10:12:32.5008696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5009019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5009450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5009799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5010136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:32.5010650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5010976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5011402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5011731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5012068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:32.5012536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5012865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5013276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5013623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5013954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:32.5014405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5014728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5015155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5015609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5015929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:32.5016479Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5017012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5017533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5018035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5018425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:32.5018776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:32.5019121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:32.5019455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:32.5019709Z dist init r=0, world=4 2022-12-01T10:12:32.5019892Z dist init r=2, world=4 2022-12-01T10:12:32.5020059Z dist init r=3, world=4 2022-12-01T10:12:32.5020238Z dist init r=1, world=4 2022-12-01T10:12:32.5020443Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:12:32.5020884Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9076 2022-12-01T10:12:32.5021366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9077 2022-12-01T10:12:32.5021731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9078 2022-12-01T10:12:32.5022057Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9079 2022-12-01T10:12:32.5022493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5022825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5023286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5023621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5023958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:32.5024428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5024757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5025167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5025517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5025853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:32.5026347Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5026833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5027162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5027590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5027922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5028258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:32.5028724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:32.5029051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:32.5029464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:32.5029813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:32.5030145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:32.5030642Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5031152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5031671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:32.5032054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:32.5032388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:32.5032736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:32.5033075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:32.5033332Z dist init r=1, world=4 2022-12-01T10:12:32.5033501Z dist init r=0, world=4 2022-12-01T10:12:32.5033681Z dist init r=2, world=4 2022-12-01T10:12:32.5033857Z dist init r=3, world=4 2022-12-01T10:12:32.5034049Z skip: Need at least 2 CUDA devices (1.416s) 2022-12-01T10:12:32.5034188Z 2022-12-01T10:12:32.5034424Z ---------------------------------------------------------------------- 2022-12-01T10:12:32.5034669Z Ran 8 tests in 11.253s 2022-12-01T10:12:32.5034783Z 2022-12-01T10:12:32.5034841Z OK (skipped=8) 2022-12-01T10:12:32.5034949Z 2022-12-01T10:12:32.5035036Z Generating XML reports... 2022-12-01T10:12:32.5035525Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221201101220.xml 2022-12-01T10:12:32.5035839Z 2022-12-01T10:12:32.5036177Z ##[endgroup] 2022-12-01T10:12:32.5036658Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_f79ssb64) 2022-12-01T10:12:32.5036938Z 2022-12-01T10:12:32.5037145Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-12-01 10:12:32.492018] 2022-12-01T10:12:32.5037677Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:12:32.492428] 2022-12-01T10:12:44.8577824Z 2022-12-01T10:12:44.8578411Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order 2022-12-01T10:12:44.8579577Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_vd1nu0ic) 2022-12-01T10:12:44.8580769Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order 2022-12-01T10:12:44.8581072Z 2022-12-01T10:12:44.8581164Z Running tests... 2022-12-01T10:12:44.8581682Z ---------------------------------------------------------------------- 2022-12-01T10:12:44.8582323Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-12-01T10:12:44.8583048Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9143 2022-12-01T10:12:44.8583444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9144 2022-12-01T10:12:44.8583774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9145 2022-12-01T10:12:44.8584096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9146 2022-12-01T10:12:44.8584632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8585071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8585699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8586127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8586510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:44.8586989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8587386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8587800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8588212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8588551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:44.8589092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8589412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8589908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8590252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8590844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:44.8591364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8591695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8592189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8592583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8592979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:44.8593482Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8594076Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8594653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8595230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8595620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:44.8596040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:44.8596376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:44.8596753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:44.8597039Z dist init r=1, world=4 2022-12-01T10:12:44.8597206Z dist init r=0, world=4 2022-12-01T10:12:44.8597384Z dist init r=2, world=4 2022-12-01T10:12:44.8597620Z dist init r=3, world=4 2022-12-01T10:12:44.8597816Z skip: Need at least 2 CUDA devices (1.650s) 2022-12-01T10:12:44.8598140Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-12-01T10:12:44.8598924Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9199 2022-12-01T10:12:44.8599399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9200 2022-12-01T10:12:44.8599719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9201 2022-12-01T10:12:44.8600102Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9202 2022-12-01T10:12:44.8600555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8601000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8601692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8602189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8602527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:44.8602990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8603324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8603753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8604098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8604419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:44.8604980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8605310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8605727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8606071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8606401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:44.8606947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8607430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8607754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8608179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8608528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8608846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:44.8609342Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8609860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8610363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8610743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:44.8611092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:44.8611437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:44.8611768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:44.8612024Z dist init r=0, world=4 2022-12-01T10:12:44.8612206Z dist init r=2, world=4 2022-12-01T10:12:44.8612370Z dist init r=3, world=4 2022-12-01T10:12:44.8612547Z dist init r=1, world=4 2022-12-01T10:12:44.8612751Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:12:44.8613087Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-12-01T10:12:44.8613663Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9255 2022-12-01T10:12:44.8614066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9256 2022-12-01T10:12:44.8614391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9257 2022-12-01T10:12:44.8614700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9258 2022-12-01T10:12:44.8615150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8615478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8615907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8616309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8616651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:44.8617123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8617438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8617905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8618252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8618588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:44.8619042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8619399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8619822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8620161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8620477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:44.8620975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8621473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8621784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8622212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8622553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8622888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:44.8623371Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8623893Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8624413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8624798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:44.8625134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:44.8625476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:44.8625824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:44.8626068Z dist init r=1, world=4 2022-12-01T10:12:44.8626251Z dist init r=2, world=4 2022-12-01T10:12:44.8626428Z dist init r=0, world=4 2022-12-01T10:12:44.8626592Z dist init r=3, world=4 2022-12-01T10:12:44.8626798Z skip: Need at least 2 CUDA devices (1.417s) 2022-12-01T10:12:44.8627143Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-12-01T10:12:44.8627716Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9311 2022-12-01T10:12:44.8628103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9312 2022-12-01T10:12:44.8628431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9313 2022-12-01T10:12:44.8628752Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9314 2022-12-01T10:12:44.8629209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8629526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8629957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8630301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8630651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:44.8631119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8631446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8631877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8632246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8632581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:44.8633046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8633359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8633789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8634135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8634469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:44.8634918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8635247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8635672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8636017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8636337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:44.8636834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8637357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8637862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8638659Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8639033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:44.8639382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:44.8639722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:44.8640068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:44.8640318Z dist init r=2, world=4 2022-12-01T10:12:44.8640501Z dist init r=1, world=4 2022-12-01T10:12:44.8640681Z dist init r=3, world=4 2022-12-01T10:12:44.8640845Z dist init r=0, world=4 2022-12-01T10:12:44.8641052Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:12:44.8641402Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-12-01T10:12:44.8641978Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9367 2022-12-01T10:12:44.8642369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9368 2022-12-01T10:12:44.8642697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9369 2022-12-01T10:12:44.8643020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9370 2022-12-01T10:12:44.8643550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8643885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8644314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8644663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8644991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:44.8645503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8645831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8646244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8646589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8646924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:44.8647388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8647701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8648126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8648469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8648801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:44.8649280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8649773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8650100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8650512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8650855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8651189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:44.8651683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8652185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8652702Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8653093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:44.8653442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:44.8653772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:44.8654108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:44.8654366Z dist init r=2, world=4 2022-12-01T10:12:44.8654533Z dist init r=0, world=4 2022-12-01T10:12:44.8654712Z dist init r=3, world=4 2022-12-01T10:12:44.8654891Z dist init r=1, world=4 2022-12-01T10:12:44.8655084Z skip: Need at least 2 CUDA devices (1.417s) 2022-12-01T10:12:44.8655434Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-12-01T10:12:44.8656005Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9423 2022-12-01T10:12:44.8656501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9424 2022-12-01T10:12:44.8656817Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9425 2022-12-01T10:12:44.8657135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9426 2022-12-01T10:12:44.8657589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8657950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8658366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8658710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8659046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:44.8659501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8659829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8660258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8660601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8660922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:44.8661390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8661719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8662131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8662474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8662808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:44.8663270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8663585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8664009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8664356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8664687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:44.8665163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8665683Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8666201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8666713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8667082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:44.8667436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:44.8667781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:44.8668109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:44.8668364Z dist init r=0, world=4 2022-12-01T10:12:44.8668548Z dist init r=2, world=4 2022-12-01T10:12:44.8668712Z dist init r=3, world=4 2022-12-01T10:12:44.8668888Z dist init r=1, world=4 2022-12-01T10:12:44.8669120Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:12:44.8669519Z test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9479 2022-12-01T10:12:44.8669930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9480 2022-12-01T10:12:44.8670256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9481 2022-12-01T10:12:44.8670623Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9482 2022-12-01T10:12:44.8671061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8671390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8671816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8672164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8672486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:44.8672951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8673280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8673691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8674041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8674375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:44.8674837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8675149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8675572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8675913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8676243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:44.8676692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8677022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8677443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8677772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8678102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:44.8678769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8679291Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8679793Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8680312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8680704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:44.8681060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:44.8681389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:44.8681779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:44.8682041Z dist init r=0, world=4 2022-12-01T10:12:44.8682211Z dist init r=3, world=4 2022-12-01T10:12:44.8682389Z dist init r=2, world=4 2022-12-01T10:12:44.8682567Z dist init r=1, world=4 2022-12-01T10:12:44.8682759Z skip: Need at least 2 CUDA devices (1.418s) 2022-12-01T10:12:44.8683160Z test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9535 2022-12-01T10:12:44.8683626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9536 2022-12-01T10:12:44.8683953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9537 2022-12-01T10:12:44.8684264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9538 2022-12-01T10:12:44.8684712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8685043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8685471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8685803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8686139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:44.8686607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8686922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8687351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8687698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8688036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:44.8688490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8688818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8689243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8689576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8689910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:12:44.8690404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8690899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:44.8691214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:44.8691644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:44.8691987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:44.8692323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:12:44.8692805Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8693321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8693832Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:12:44.8694214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:44.8694577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:12:44.8694917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:44.8695261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:12:44.8695509Z dist init r=1, world=4 2022-12-01T10:12:44.8695690Z dist init r=0, world=4 2022-12-01T10:12:44.8695867Z dist init r=3, world=4 2022-12-01T10:12:44.8696060Z dist init r=2, world=4 2022-12-01T10:12:44.8696317Z skip: Need at least 2 CUDA devices (1.318s) 2022-12-01T10:12:44.8696458Z 2022-12-01T10:12:44.8696664Z ---------------------------------------------------------------------- 2022-12-01T10:12:44.8696891Z Ran 8 tests in 11.171s 2022-12-01T10:12:44.8697004Z 2022-12-01T10:12:44.8697077Z OK (skipped=8) 2022-12-01T10:12:44.8697186Z 2022-12-01T10:12:44.8697272Z Generating XML reports... 2022-12-01T10:12:44.8697735Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221201101233.xml 2022-12-01T10:12:44.8698004Z 2022-12-01T10:12:44.8698313Z ##[endgroup] 2022-12-01T10:12:44.8698783Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_vd1nu0ic) 2022-12-01T10:12:44.8699053Z 2022-12-01T10:12:44.8699246Z Running distributed/fsdp/test_fsdp_meta ... [2022-12-01 10:12:44.858397] 2022-12-01T10:12:44.8699749Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:12:44.860032] 2022-12-01T10:12:54.7540778Z 2022-12-01T10:12:54.7541478Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta 2022-12-01T10:12:54.7542502Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_1u1igqz1) 2022-12-01T10:12:54.7543174Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta 2022-12-01T10:12:54.7543381Z 2022-12-01T10:12:54.7543526Z Running tests... 2022-12-01T10:12:54.7543889Z ---------------------------------------------------------------------- 2022-12-01T10:12:54.7544297Z test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9602 2022-12-01T10:12:54.7544874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9603 2022-12-01T10:12:54.7545420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7545761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7546251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7546600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7546984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:54.7547513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7547834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7548322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7548679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7549071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:54.7549620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7550384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7550835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:54.7551205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:54.7551480Z dist init r=0, world=2 2022-12-01T10:12:54.7551670Z dist init r=1, world=2 2022-12-01T10:12:54.7551882Z skip: Need at least 2 CUDA devices (1.439s) 2022-12-01T10:12:54.7552306Z test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-12-01T10:12:54.7553020Z test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9630 2022-12-01T10:12:54.7553515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9631 2022-12-01T10:12:54.7554013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7554381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7554822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7555217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7555547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:54.7556078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7556411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7556893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7557225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7557623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:54.7558168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7558925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7559316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:54.7559715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:54.7559993Z dist init r=1, world=2 2022-12-01T10:12:54.7560162Z dist init r=0, world=2 2022-12-01T10:12:54.7560371Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:12:54.7560839Z test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9658 2022-12-01T10:12:54.7561328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9659 2022-12-01T10:12:54.7561777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7562174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7562608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7563001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7563341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:54.7563886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7564215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7564771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7565123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7565528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:54.7566094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7566651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7567111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:54.7567464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:54.7567782Z dist init r=0, world=2 2022-12-01T10:12:54.7567967Z dist init r=1, world=2 2022-12-01T10:12:54.7568189Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:12:54.7568637Z test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9686 2022-12-01T10:12:54.7569071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9687 2022-12-01T10:12:54.7569596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7569963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7570409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7570822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7571165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:54.7571707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7572026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7572523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7572869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7573254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:54.7573754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7574346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7574803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:54.7575140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:54.7575394Z dist init r=1, world=2 2022-12-01T10:12:54.7575641Z dist init r=0, world=2 2022-12-01T10:12:54.7575840Z skip: Need at least 2 CUDA devices (1.213s) 2022-12-01T10:12:54.7576327Z test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9714 2022-12-01T10:12:54.7576776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9715 2022-12-01T10:12:54.7577330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7577648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7578094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7578539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7578887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:54.7579348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7579742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7580172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7580539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7580875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:54.7581376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7581970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7582349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:54.7582698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:54.7582961Z dist init r=1, world=2 2022-12-01T10:12:54.7583129Z dist init r=0, world=2 2022-12-01T10:12:54.7583340Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:12:54.7583755Z test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-12-01T10:12:54.7584304Z test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-12-01T10:12:54.7584829Z test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-12-01T10:12:54.7585359Z test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-12-01T10:12:54.7585952Z test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9742 2022-12-01T10:12:54.7586372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9743 2022-12-01T10:12:54.7586833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7587152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7587584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7587937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7588262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:54.7588731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7589059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7589484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7589818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7590154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:54.7590652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7591203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7591592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:54.7591940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:54.7592204Z dist init r=1, world=2 2022-12-01T10:12:54.7592372Z dist init r=0, world=2 2022-12-01T10:12:54.7592653Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:12:54.7593076Z test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9770 2022-12-01T10:12:54.7593482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9771 2022-12-01T10:12:54.7593942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7594276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7594706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7595038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7595378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:12:54.7595848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:12:54.7596183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:12:54.7596600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:12:54.7596946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:12:54.7597286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:12:54.7597771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7598383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:12:54.7598773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:12:54.7599125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:12:54.7599375Z dist init r=0, world=2 2022-12-01T10:12:54.7599557Z dist init r=1, world=2 2022-12-01T10:12:54.7599770Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:12:54.7600147Z test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-12-01T10:12:54.7600663Z test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-12-01T10:12:54.7600930Z 2022-12-01T10:12:54.7601132Z ---------------------------------------------------------------------- 2022-12-01T10:12:54.7601377Z Ran 14 tests in 8.706s 2022-12-01T10:12:54.7601492Z 2022-12-01T10:12:54.7601553Z OK (skipped=14) 2022-12-01T10:12:54.7601663Z 2022-12-01T10:12:54.7601748Z Generating XML reports... 2022-12-01T10:12:54.7602211Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221201101245.xml 2022-12-01T10:12:54.7602484Z 2022-12-01T10:12:54.7603918Z ##[endgroup] 2022-12-01T10:12:54.7604403Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_1u1igqz1) 2022-12-01T10:12:54.7604661Z 2022-12-01T10:12:54.7604877Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-12-01 10:12:54.754793] 2022-12-01T10:12:54.7605505Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:12:54.755153] 2022-12-01T10:13:02.8400521Z 2022-12-01T10:13:02.8401285Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_ignored_modules 2022-12-01T10:13:02.8402430Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_irgy1srr) 2022-12-01T10:13:02.8403417Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules 2022-12-01T10:13:02.8403642Z 2022-12-01T10:13:02.8403703Z Running tests... 2022-12-01T10:13:02.8404024Z ---------------------------------------------------------------------- 2022-12-01T10:13:02.8404388Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules) 2022-12-01T10:13:02.8404815Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9809 2022-12-01T10:13:02.8405184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9810 2022-12-01T10:13:02.8405515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9811 2022-12-01T10:13:02.8405842Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9812 2022-12-01T10:13:02.8406319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8406645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8407078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8435029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8435686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:02.8436327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8436676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8437106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8437459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8437802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:02.8438447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8442232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8443048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8443640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8443982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:13:02.8444662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8445501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8446130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8446974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8447594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8447958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:13:02.8448586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8449111Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8449617Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8450003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:02.8450411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:02.8450754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:02.8451086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:02.8451400Z dist init r=2, world=4 2022-12-01T10:13:02.8451587Z dist init r=1, world=4 2022-12-01T10:13:02.8451757Z dist init r=0, world=4 2022-12-01T10:13:02.8451937Z dist init r=3, world=4 2022-12-01T10:13:02.8452142Z skip: Need at least 2 CUDA devices (1.637s) 2022-12-01T10:13:02.8452449Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules) 2022-12-01T10:13:02.8452864Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9865 2022-12-01T10:13:02.8453241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9866 2022-12-01T10:13:02.8453568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9867 2022-12-01T10:13:02.8453878Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9868 2022-12-01T10:13:02.8454337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8454778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8455560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8455901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8456298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:02.8456778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8457111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8457848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8458472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8459085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:02.8459947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8460502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8460933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8461276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8461597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:13:02.8462347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8462885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8463672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8464390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8464970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:13:02.8465494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8466262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8467237Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8467763Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8468158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:02.8468511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:02.8468846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:02.8469185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:02.8469446Z dist init r=0, world=4 2022-12-01T10:13:02.8469616Z dist init r=1, world=4 2022-12-01T10:13:02.8469793Z dist init r=3, world=4 2022-12-01T10:13:02.8469972Z dist init r=2, world=4 2022-12-01T10:13:02.8470180Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:13:02.8470441Z test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules) 2022-12-01T10:13:02.8470826Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9921 2022-12-01T10:13:02.8471217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9922 2022-12-01T10:13:02.8471542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9923 2022-12-01T10:13:02.8471857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9924 2022-12-01T10:13:02.8472302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8472631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8473044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8473394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8473735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:02.8474206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8474521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8474947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8475295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8475618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:13:02.8476108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8486965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8487478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8488241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8488867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8489632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:02.8490510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8491102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8491885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8492307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8492696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:13:02.8493183Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8493705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8494224Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8494608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:02.8494939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:02.8495280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:02.8495630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:02.8495876Z dist init r=3, world=4 2022-12-01T10:13:02.8496056Z dist init r=2, world=4 2022-12-01T10:13:02.8496308Z dist init r=0, world=4 2022-12-01T10:13:02.8496473Z dist init r=1, world=4 2022-12-01T10:13:02.8496683Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:13:02.8496953Z test_ignored_modules_nested (__main__.TestFSDPIgnoredModules) 2022-12-01T10:13:02.8497346Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9977 2022-12-01T10:13:02.8497726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9978 2022-12-01T10:13:02.8498058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9979 2022-12-01T10:13:02.8498387Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9980 2022-12-01T10:13:02.8498828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8499163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8499591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8499935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8500257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:02.8500762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8501088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8501502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8501844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8502182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:02.8502648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8502961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8503383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8503773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8504107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:13:02.8504556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8504883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8505342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8505675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8506007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:13:02.8506500Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8507064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8507587Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8508080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8508467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:02.8508821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:02.8509152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:02.8509497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:02.8509751Z dist init r=3, world=4 2022-12-01T10:13:02.8509934Z dist init r=2, world=4 2022-12-01T10:13:02.8510099Z dist init r=1, world=4 2022-12-01T10:13:02.8510280Z dist init r=0, world=4 2022-12-01T10:13:02.8510485Z skip: Need at least 2 CUDA devices (1.316s) 2022-12-01T10:13:02.8510746Z test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules) 2022-12-01T10:13:02.8511245Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10033 2022-12-01T10:13:02.8511639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10034 2022-12-01T10:13:02.8511958Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10035 2022-12-01T10:13:02.8512280Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10036 2022-12-01T10:13:02.8512727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8513057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8513473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8513819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8514152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:13:02.8514614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8514930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8515349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8515688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8516005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:02.8516528Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8517027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8517351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8517762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8518138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8518661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:02.8519142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:02.8519457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:02.8519890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:02.8520234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:02.8520552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:13:02.8521043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8521566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8522077Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:02.8522446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:02.8522793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:02.8523137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:02.8523480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:02.8523724Z dist init r=0, world=4 2022-12-01T10:13:02.8523905Z dist init r=3, world=4 2022-12-01T10:13:02.8524081Z dist init r=2, world=4 2022-12-01T10:13:02.8524242Z dist init r=1, world=4 2022-12-01T10:13:02.8524445Z skip: Need at least 2 CUDA devices (1.317s) 2022-12-01T10:13:02.8524586Z 2022-12-01T10:13:02.8524789Z ---------------------------------------------------------------------- 2022-12-01T10:13:02.8525020Z Ran 5 tests in 6.903s 2022-12-01T10:13:02.8525132Z 2022-12-01T10:13:02.8525204Z OK (skipped=5) 2022-12-01T10:13:02.8525311Z 2022-12-01T10:13:02.8525394Z Generating XML reports... 2022-12-01T10:13:02.8525865Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221201101255.xml 2022-12-01T10:13:02.8526150Z 2022-12-01T10:13:02.8526536Z ##[endgroup] 2022-12-01T10:13:02.8527020Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_irgy1srr) 2022-12-01T10:13:02.8527304Z 2022-12-01T10:13:02.8527502Z Running distributed/fsdp/test_fsdp_apply ... [2022-12-01 10:13:02.841155] 2022-12-01T10:13:02.8528006Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:02.841482] 2022-12-01T10:13:07.8638380Z 2022-12-01T10:13:07.8638893Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply 2022-12-01T10:13:07.8640083Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_rhwjg6ov) 2022-12-01T10:13:07.8641155Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply 2022-12-01T10:13:07.8641797Z 2022-12-01T10:13:07.8641938Z Running tests... 2022-12-01T10:13:07.8642493Z ---------------------------------------------------------------------- 2022-12-01T10:13:07.8642994Z test_apply_in_summon_raises_error (__main__.TestApply) 2022-12-01T10:13:07.8643765Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10100 2022-12-01T10:13:07.8644488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10101 2022-12-01T10:13:07.8645447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:07.8646058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:07.8646864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:07.8647487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:07.8647883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:07.8648616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:07.8649188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:07.8649616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:07.8649955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:07.8650291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:07.8650786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:07.8651308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:07.8651682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:07.8652031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:07.8652291Z dist init r=1, world=2 2022-12-01T10:13:07.8652461Z dist init r=0, world=2 2022-12-01T10:13:07.8652668Z skip: Need at least 2 CUDA devices (1.432s) 2022-12-01T10:13:07.8652912Z test_nested_module_apply (__main__.TestApply) 2022-12-01T10:13:07.8653367Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10128 2022-12-01T10:13:07.8653769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10129 2022-12-01T10:13:07.8654224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:07.8654558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:07.8654973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:07.8655318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:07.8655654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:07.8656191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:07.8656511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:07.8656938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:07.8657281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:07.8657604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:07.8658159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:07.8658679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:07.8659066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:07.8659399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:07.8659691Z dist init r=1, world=2 2022-12-01T10:13:07.8659872Z dist init r=0, world=2 2022-12-01T10:13:07.8660067Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:13:07.8660317Z test_transformer_module_apply (__main__.TestApply) 2022-12-01T10:13:07.8660795Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10156 2022-12-01T10:13:07.8661197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10157 2022-12-01T10:13:07.8661636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:07.8661966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:07.8662392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:07.8662729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:07.8663064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:07.8663532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:07.8663859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:07.8664274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:07.8664618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:07.8664953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:07.8665448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:07.8665956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:07.8666347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:07.8666694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:07.8666939Z dist init r=0, world=2 2022-12-01T10:13:07.8667119Z dist init r=1, world=2 2022-12-01T10:13:07.8667329Z skip: Need at least 2 CUDA devices (1.210s) 2022-12-01T10:13:07.8667468Z 2022-12-01T10:13:07.8667673Z ---------------------------------------------------------------------- 2022-12-01T10:13:07.8667903Z Ran 3 tests in 3.852s 2022-12-01T10:13:07.8668017Z 2022-12-01T10:13:07.8668089Z OK (skipped=3) 2022-12-01T10:13:07.8668196Z 2022-12-01T10:13:07.8668282Z Generating XML reports... 2022-12-01T10:13:07.8668699Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221201101303.xml 2022-12-01T10:13:07.8668945Z 2022-12-01T10:13:07.8669220Z ##[endgroup] 2022-12-01T10:13:07.8669674Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_rhwjg6ov) 2022-12-01T10:13:07.8669937Z 2022-12-01T10:13:07.8670119Z Running distributed/fsdp/test_fsdp_memory ... [2022-12-01 10:13:07.864248] 2022-12-01T10:13:07.8670693Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:07.864677] 2022-12-01T10:13:11.6902429Z 2022-12-01T10:13:11.6902994Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory 2022-12-01T10:13:11.6904182Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_c_13o8na) 2022-12-01T10:13:11.6904935Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory 2022-12-01T10:13:11.6905358Z 2022-12-01T10:13:11.6905474Z Running tests... 2022-12-01T10:13:11.6905812Z ---------------------------------------------------------------------- 2022-12-01T10:13:11.6906426Z test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10195 2022-12-01T10:13:11.6907130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10196 2022-12-01T10:13:11.6907963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:11.6908565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:11.6909344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:11.6909783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:11.6910123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:11.6910690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:11.6911027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:11.6911456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:11.6911874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:11.6912484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:11.6913088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:11.6913616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:11.6914005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:11.6914345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:11.6914601Z dist init r=0, world=2 2022-12-01T10:13:11.6914785Z dist init r=1, world=2 2022-12-01T10:13:11.6914984Z skip: Need at least 2 CUDA devices (1.434s) 2022-12-01T10:13:11.6915333Z test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10223 2022-12-01T10:13:11.6915725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10224 2022-12-01T10:13:11.6916168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:11.6916500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:11.6916929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:11.6917281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:11.6917603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:11.6918071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:11.6918540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:11.6919072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:11.6919404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:11.6919739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:11.6920236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:11.6920850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:11.6921226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:11.6921578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:11.6921838Z dist init r=0, world=2 2022-12-01T10:13:11.6922006Z dist init r=1, world=2 2022-12-01T10:13:11.6922213Z skip: Need at least 2 CUDA devices (1.222s) 2022-12-01T10:13:11.6922352Z 2022-12-01T10:13:11.6922555Z ---------------------------------------------------------------------- 2022-12-01T10:13:11.6922799Z Ran 2 tests in 2.656s 2022-12-01T10:13:11.6922899Z 2022-12-01T10:13:11.6922972Z OK (skipped=2) 2022-12-01T10:13:11.6923079Z 2022-12-01T10:13:11.6923166Z Generating XML reports... 2022-12-01T10:13:11.6923613Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221201101308.xml 2022-12-01T10:13:11.6923867Z 2022-12-01T10:13:11.6924121Z ##[endgroup] 2022-12-01T10:13:11.6924573Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_c_13o8na) 2022-12-01T10:13:11.6924830Z 2022-12-01T10:13:11.6925025Z Running distributed/fsdp/test_fsdp_input ... [2022-12-01 10:13:11.690587] 2022-12-01T10:13:11.6925527Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:11.691002] 2022-12-01T10:13:15.3117412Z 2022-12-01T10:13:15.3117974Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input 2022-12-01T10:13:15.3119251Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_4ukzmfz7) 2022-12-01T10:13:15.3119891Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input 2022-12-01T10:13:15.3120114Z 2022-12-01T10:13:15.3120190Z Running tests... 2022-12-01T10:13:15.3120504Z ---------------------------------------------------------------------- 2022-12-01T10:13:15.3120781Z test_input_type_dict (__main__.TestInput) 2022-12-01T10:13:15.3121125Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10262 2022-12-01T10:13:15.3121663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:15.3122000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:15.3122543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:15.3122940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:15.3123293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:15.3123808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:13:15.3124188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:15.3124456Z dist init r=0, world=1 2022-12-01T10:13:15.3124666Z skip: Need at least 1 CUDA device (1.336s) 2022-12-01T10:13:15.3124886Z test_input_type_list (__main__.TestInput) 2022-12-01T10:13:15.3125503Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10276 2022-12-01T10:13:15.3126108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:15.3126505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:15.3126928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:15.3127351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:15.3127694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:15.3128186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-12-01T10:13:15.3128577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:15.3128844Z dist init r=0, world=1 2022-12-01T10:13:15.3129059Z skip: Need at least 1 CUDA device (1.107s) 2022-12-01T10:13:15.3129182Z 2022-12-01T10:13:15.3129389Z ---------------------------------------------------------------------- 2022-12-01T10:13:15.3129636Z Ran 2 tests in 2.444s 2022-12-01T10:13:15.3129753Z 2022-12-01T10:13:15.3129826Z OK (skipped=2) 2022-12-01T10:13:15.3129936Z 2022-12-01T10:13:15.3130009Z Generating XML reports... 2022-12-01T10:13:15.3130441Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221201101312.xml 2022-12-01T10:13:15.3130684Z 2022-12-01T10:13:15.3130946Z ##[endgroup] 2022-12-01T10:13:15.3131384Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_4ukzmfz7) 2022-12-01T10:13:15.3131644Z 2022-12-01T10:13:15.3131864Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-12-01 10:13:15.312093] 2022-12-01T10:13:15.3132391Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:15.312452] 2022-12-01T10:13:18.1318103Z 2022-12-01T10:13:18.1318869Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_pure_fp16 2022-12-01T10:13:18.1319982Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_vx8k_gz8) 2022-12-01T10:13:18.1320671Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16 2022-12-01T10:13:18.1320886Z 2022-12-01T10:13:18.1320962Z Running tests... 2022-12-01T10:13:18.1321271Z ---------------------------------------------------------------------- 2022-12-01T10:13:18.1321592Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-12-01T10:13:18.1322658Z Tests pure FP16 training, including when the parameter's dtype is ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/73315 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.220s) 2022-12-01T10:13:18.1323354Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16) 2022-12-01T10:13:18.1323880Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10301 2022-12-01T10:13:18.1324295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10302 2022-12-01T10:13:18.1324619Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10303 2022-12-01T10:13:18.1324989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10304 2022-12-01T10:13:18.1325777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:18.1326187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:18.1326616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:18.1326969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:18.1327313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:18.1327871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:18.1328202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:18.1328632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:18.1328981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:18.1329312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:13:18.1329786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:18.1330118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:18.1330533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:18.1330880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:18.1331220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:18.1331689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:18.1332005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:18.1332437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:18.1332786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:18.1333121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:13:18.1333604Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:18.1334129Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:18.1334648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:18.1335164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:18.1335539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:18.1335889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:18.1336298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:18.1336636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:18.1336899Z dist init r=2, world=4 2022-12-01T10:13:18.1337088Z dist init r=1, world=4 2022-12-01T10:13:18.1337254Z dist init r=3, world=4 2022-12-01T10:13:18.1337436Z dist init r=0, world=4 2022-12-01T10:13:18.1337645Z skip: Need at least 2 CUDA devices (1.420s) 2022-12-01T10:13:18.1337785Z 2022-12-01T10:13:18.1337990Z ---------------------------------------------------------------------- 2022-12-01T10:13:18.1338222Z Ran 2 tests in 1.640s 2022-12-01T10:13:18.1338338Z 2022-12-01T10:13:18.1338412Z OK (skipped=2) 2022-12-01T10:13:18.1338520Z 2022-12-01T10:13:18.1338608Z Generating XML reports... 2022-12-01T10:13:18.1339091Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221201101316.xml 2022-12-01T10:13:18.1339344Z 2022-12-01T10:13:18.1339620Z ##[endgroup] 2022-12-01T10:13:18.1340083Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_vx8k_gz8) 2022-12-01T10:13:18.1340342Z 2022-12-01T10:13:18.1340558Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-12-01 10:13:18.132107] 2022-12-01T10:13:18.1341148Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:18.132444] 2022-12-01T10:13:20.9487565Z 2022-12-01T10:13:20.9488133Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping 2022-12-01T10:13:20.9489130Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_n8a1pxhd) 2022-12-01T10:13:20.9489837Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping 2022-12-01T10:13:20.9490059Z 2022-12-01T10:13:20.9490135Z Running tests... 2022-12-01T10:13:20.9490431Z ---------------------------------------------------------------------- 2022-12-01T10:13:20.9490732Z test_multiple_wrapping (__main__.TestMultipleWrapping) 2022-12-01T10:13:20.9491252Z This test simulates wrapping the module after training to run inference. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10368 2022-12-01T10:13:20.9491730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10369 2022-12-01T10:13:20.9492188Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10370 2022-12-01T10:13:20.9492619Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10371 2022-12-01T10:13:20.9493108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:20.9493430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:20.9493870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:20.9494221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:20.9494554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:20.9495066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:20.9495479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:20.9496021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:20.9496472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:20.9496813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:20.9497349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:20.9497704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:20.9498149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:20.9498558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:20.9498897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:13:20.9499400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:20.9499922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:20.9500400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:20.9500801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:20.9501125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:13:20.9501675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:20.9502325Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:20.9502834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:20.9503409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:20.9503855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:20.9504211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:20.9504598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:20.9504940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:20.9505211Z dist init r=1, world=4 2022-12-01T10:13:20.9505432Z dist init r=3, world=4 2022-12-01T10:13:20.9505616Z dist init r=2, world=4 2022-12-01T10:13:20.9505796Z dist init r=0, world=4 2022-12-01T10:13:20.9506016Z skip: Need at least 2 CUDA devices (1.638s) 2022-12-01T10:13:20.9506181Z 2022-12-01T10:13:20.9506386Z ---------------------------------------------------------------------- 2022-12-01T10:13:20.9506633Z Ran 1 test in 1.638s 2022-12-01T10:13:20.9506749Z 2022-12-01T10:13:20.9506865Z OK (skipped=1) 2022-12-01T10:13:20.9506979Z 2022-12-01T10:13:20.9507054Z Generating XML reports... 2022-12-01T10:13:20.9507565Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221201101319.xml 2022-12-01T10:13:20.9507866Z 2022-12-01T10:13:20.9508145Z ##[endgroup] 2022-12-01T10:13:20.9508681Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_n8a1pxhd) 2022-12-01T10:13:20.9508974Z 2022-12-01T10:13:20.9509266Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-12-01 10:13:20.949047] 2022-12-01T10:13:20.9509863Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:20.949451] 2022-12-01T10:13:23.7813639Z 2022-12-01T10:13:23.7814466Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec 2022-12-01T10:13:23.7815784Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_8xgk_78v) 2022-12-01T10:13:23.7816797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxep9tt93 2022-12-01T10:13:23.7817527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxep9tt93/_remote_module_non_scriptable.py 2022-12-01T10:13:23.7818346Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec 2022-12-01T10:13:23.7818635Z 2022-12-01T10:13:23.7818741Z Running tests... 2022-12-01T10:13:23.7819076Z ---------------------------------------------------------------------- 2022-12-01T10:13:23.7819482Z test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10435 2022-12-01T10:13:23.7820069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10436 2022-12-01T10:13:23.7820406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10437 2022-12-01T10:13:23.7820726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10438 2022-12-01T10:13:23.7821198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:23.7821545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:23.7822251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:23.7822592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:23.7822943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5vdogjs 2022-12-01T10:13:23.7823352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5vdogjs/_remote_module_non_scriptable.py 2022-12-01T10:13:23.7823721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:23.7824188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:23.7824522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:23.7824950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:23.7825289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:23.7825635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsps6a9x7 2022-12-01T10:13:23.7826038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsps6a9x7/_remote_module_non_scriptable.py 2022-12-01T10:13:23.7826509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:23.7826841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:23.7827268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:23.7827615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:23.7827946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9g72px8 2022-12-01T10:13:23.7828345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9g72px8/_remote_module_non_scriptable.py 2022-12-01T10:13:23.7828724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:23.7829072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:23.7829516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:23.7829852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:23.7830280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:23.7830611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:23.7830957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8efyo4h 2022-12-01T10:13:23.7831356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8efyo4h/_remote_module_non_scriptable.py 2022-12-01T10:13:23.7831733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:23.7831972Z ok (1.642s) 2022-12-01T10:13:23.7832221Z test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec) 2022-12-01T10:13:23.7832564Z Test custom spec can be invoked from the ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:13:23.7832884Z test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec) 2022-12-01T10:13:23.7833298Z Test sharded_tensor.ones(...) with the custom ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:13:23.7833660Z test_chunked_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s) 2022-12-01T10:13:23.7834005Z test_device_placement (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s) 2022-12-01T10:13:23.7834339Z test_enumerable_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.004s) 2022-12-01T10:13:23.7834706Z test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s) 2022-12-01T10:13:23.7835004Z test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s) 2022-12-01T10:13:23.7835275Z test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s) 2022-12-01T10:13:23.7835584Z test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.007s) 2022-12-01T10:13:23.7835769Z 2022-12-01T10:13:23.7835978Z ---------------------------------------------------------------------- 2022-12-01T10:13:23.7836222Z Ran 10 tests in 1.661s 2022-12-01T10:13:23.7836324Z 2022-12-01T10:13:23.7836396Z OK (skipped=5) 2022-12-01T10:13:23.7836502Z 2022-12-01T10:13:23.7836587Z Generating XML reports... 2022-12-01T10:13:23.7837080Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221201101321.xml 2022-12-01T10:13:23.7837718Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221201101321.xml 2022-12-01T10:13:23.7837974Z 2022-12-01T10:13:23.7838481Z ##[endgroup] 2022-12-01T10:13:23.7838996Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_8xgk_78v) 2022-12-01T10:13:23.7839292Z 2022-12-01T10:13:23.7839519Z Running distributed/fsdp/test_fsdp_traversal ... [2022-12-01 10:13:23.781747] 2022-12-01T10:13:23.7840039Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:23.782103] 2022-12-01T10:13:26.4044592Z 2022-12-01T10:13:26.4045271Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal 2022-12-01T10:13:26.4046393Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_36bdf9e9) 2022-12-01T10:13:26.4047527Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal 2022-12-01T10:13:26.4047915Z 2022-12-01T10:13:26.4048046Z Running tests... 2022-12-01T10:13:26.4048468Z ---------------------------------------------------------------------- 2022-12-01T10:13:26.4048971Z test_fsdp_modules (__main__.TestTraversal) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10490 2022-12-01T10:13:26.4049489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10491 2022-12-01T10:13:26.4050329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:26.4050865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:26.4051666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:26.4052150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:26.4052490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:26.4053303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:26.4053741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:26.4054286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:26.4054620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:26.4054960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:26.4055467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:26.4056124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:26.4056501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:26.4056855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:26.4057118Z dist init r=1, world=2 2022-12-01T10:13:26.4057288Z dist init r=0, world=2 2022-12-01T10:13:26.4057500Z skip: Need at least 2 CUDA devices (1.433s) 2022-12-01T10:13:26.4057640Z 2022-12-01T10:13:26.4057845Z ---------------------------------------------------------------------- 2022-12-01T10:13:26.4058086Z Ran 1 test in 1.433s 2022-12-01T10:13:26.4058186Z 2022-12-01T10:13:26.4058260Z OK (skipped=1) 2022-12-01T10:13:26.4058369Z 2022-12-01T10:13:26.4058455Z Generating XML reports... 2022-12-01T10:13:26.4058901Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221201101324.xml 2022-12-01T10:13:26.4059160Z 2022-12-01T10:13:26.4059423Z ##[endgroup] 2022-12-01T10:13:26.4059895Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_36bdf9e9) 2022-12-01T10:13:26.4060165Z 2022-12-01T10:13:26.4060372Z Running distributed/fsdp/test_fsdp_fx ... [2022-12-01 10:13:26.404742] 2022-12-01T10:13:26.4061118Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:26.405094] 2022-12-01T10:13:29.2186486Z 2022-12-01T10:13:29.2187001Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx 2022-12-01T10:13:29.2188117Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_atfdnh9e) 2022-12-01T10:13:29.2189148Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx 2022-12-01T10:13:29.2189544Z 2022-12-01T10:13:29.2189679Z Running tests... 2022-12-01T10:13:29.2190219Z ---------------------------------------------------------------------- 2022-12-01T10:13:29.2190730Z test_symbolic_tracing_outputs (__main__.TestSymbolicTracing) 2022-12-01T10:13:29.2191417Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10529 2022-12-01T10:13:29.2192161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10530 2022-12-01T10:13:29.2192711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10531 2022-12-01T10:13:29.2193285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10532 2022-12-01T10:13:29.2194163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:29.2194760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:29.2195437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:29.2195868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:29.2196217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:29.2196675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:29.2197201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:29.2197633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:29.2198070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:29.2198828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:29.2199496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:29.2199831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:29.2200248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:29.2200597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:29.2200940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:13:29.2201439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:29.2201928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:29.2202255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:29.2202692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:29.2203027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:29.2203366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:13:29.2203864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:29.2204388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:29.2204891Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:13:29.2205280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:29.2205656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:29.2205999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:29.2206332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:29.2206597Z dist init r=0, world=4 2022-12-01T10:13:29.2206784Z dist init r=2, world=4 2022-12-01T10:13:29.2206949Z dist init r=3, world=4 2022-12-01T10:13:29.2207129Z dist init r=1, world=4 2022-12-01T10:13:29.2207300Z ok (1.641s) 2022-12-01T10:13:29.2207405Z 2022-12-01T10:13:29.2207613Z ---------------------------------------------------------------------- 2022-12-01T10:13:29.2207842Z Ran 1 test in 1.642s 2022-12-01T10:13:29.2207956Z 2022-12-01T10:13:29.2208018Z OK 2022-12-01T10:13:29.2208110Z 2022-12-01T10:13:29.2208197Z Generating XML reports... 2022-12-01T10:13:29.2208639Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221201101327.xml 2022-12-01T10:13:29.2208900Z 2022-12-01T10:13:29.2209176Z ##[endgroup] 2022-12-01T10:13:29.2209619Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_atfdnh9e) 2022-12-01T10:13:29.2209870Z 2022-12-01T10:13:29.2210078Z Running distributed/_shard/checkpoint/test_checkpoint ... [2022-12-01 10:13:29.218819] 2022-12-01T10:13:29.2210705Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:29.219440] 2022-12-01T10:13:33.3543917Z 2022-12-01T10:13:33.3544524Z Expand the folded group to see the log file of distributed/_shard/checkpoint/test_checkpoint 2022-12-01T10:13:33.3546030Z ##[group]PRINTING LOG FILE of distributed/_shard/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_checkpoint_0dg8c5s_) 2022-12-01T10:13:33.3546989Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint 2022-12-01T10:13:33.3547459Z 2022-12-01T10:13:33.3547569Z Running tests... 2022-12-01T10:13:33.3547935Z ---------------------------------------------------------------------- 2022-12-01T10:13:33.3548333Z test_default_metadata (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:13:33.3548884Z test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:13:33.3549685Z test_dummy_reader_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:13:33.3550113Z test_dummy_writer_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:13:33.3550497Z test_load_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:13:33.3551061Z test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10596 2022-12-01T10:13:33.3551467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10597 2022-12-01T10:13:33.3551854Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10598 2022-12-01T10:13:33.3552449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10599 2022-12-01T10:13:33.3553027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:33.3553368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:33.3553805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:33.3554140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:33.3554469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:33.3554934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:33.3555253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:33.3555683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:33.3556032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:33.3556356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:33.3556805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:33.3557135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:33.3557561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:33.3557895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:33.3558499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:33.3558999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:33.3559332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:33.3559858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:33.3560208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:33.3560534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:33.3560776Z ok (1.639s) 2022-12-01T10:13:33.3561074Z test_save_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:13:33.3561573Z test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10640 2022-12-01T10:13:33.3561972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10641 2022-12-01T10:13:33.3562292Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10642 2022-12-01T10:13:33.3562617Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10643 2022-12-01T10:13:33.3563070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:33.3563400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:33.3563819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:33.3564169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:33.3564496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:13:33.3564940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:33.3565273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:33.3565701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:33.3566052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:33.3566362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:13:33.3566821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:33.3567151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:33.3567567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:33.3567911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:33.3568237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:33.3568697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:33.3569015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:33.3569443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:33.3569787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:33.3570110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:33.3570347Z ok (1.316s) 2022-12-01T10:13:33.3570452Z 2022-12-01T10:13:33.3570656Z ---------------------------------------------------------------------- 2022-12-01T10:13:33.3570898Z Ran 8 tests in 2.960s 2022-12-01T10:13:33.3571013Z 2022-12-01T10:13:33.3571073Z OK (skipped=6) 2022-12-01T10:13:33.3571182Z 2022-12-01T10:13:33.3571268Z Generating XML reports... 2022-12-01T10:13:33.3571755Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221201101330.xml 2022-12-01T10:13:33.3572455Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221201101330.xml 2022-12-01T10:13:33.3572743Z 2022-12-01T10:13:33.3573024Z ##[endgroup] 2022-12-01T10:13:33.3573522Z FINISHED PRINTING LOG FILE of distributed/_shard/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_checkpoint_0dg8c5s_) 2022-12-01T10:13:33.3573813Z 2022-12-01T10:13:33.3574075Z Running distributed/_shard/checkpoint/test_file_system_checkpoint_cpu ... [2022-12-01 10:13:33.354696] 2022-12-01T10:13:33.3574685Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:33.355032] 2022-12-01T10:13:41.3255493Z 2022-12-01T10:13:41.3256269Z Expand the folded group to see the log file of distributed/_shard/checkpoint/test_file_system_checkpoint_cpu 2022-12-01T10:13:41.3257648Z ##[group]PRINTING LOG FILE of distributed/_shard/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_file_system_checkpoint_cpu__za4oypi) 2022-12-01T10:13:41.3258968Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu 2022-12-01T10:13:41.3259417Z 2022-12-01T10:13:41.3259553Z Running tests... 2022-12-01T10:13:41.3260092Z ---------------------------------------------------------------------- 2022-12-01T10:13:41.3260752Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10695 2022-12-01T10:13:41.3261308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10696 2022-12-01T10:13:41.3261877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3262216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3262659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3263013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3263328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:41.3263804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3264291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3264827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3265275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3265601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:41.3265968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:41.3266325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:41.3266825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3267347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3267638Z ok (1.534s) 2022-12-01T10:13:41.3267977Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10723 2022-12-01T10:13:41.3268397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10724 2022-12-01T10:13:41.3268854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3269359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3269793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3270141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3270470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:41.3270917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3271313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3271747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3272093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3272403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:41.3272768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:41.3273138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:41.3273621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3274145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3274438Z ok (1.412s) 2022-12-01T10:13:41.3274776Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10751 2022-12-01T10:13:41.3275169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10752 2022-12-01T10:13:41.3275625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3275957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3276384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3276717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3277041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:41.3277502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3277816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3278468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3278817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3279145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:41.3279496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:41.3279866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:41.3280366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3280877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3281167Z ok (1.210s) 2022-12-01T10:13:41.3281526Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10779 2022-12-01T10:13:41.3281953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10780 2022-12-01T10:13:41.3282457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3282789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3283218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3283565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3283924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:41.3284384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3284716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3285129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3285479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3285801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:41.3286163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:41.3286515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:41.3287008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3287535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3287826Z ok (1.412s) 2022-12-01T10:13:41.3288080Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.019s) 2022-12-01T10:13:41.3288575Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10807 2022-12-01T10:13:41.3289038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10808 2022-12-01T10:13:41.3289476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3289805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3290233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3290582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3290890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:13:41.3291346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:13:41.3291673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:13:41.3292086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:13:41.3292431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:13:41.3292753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:13:41.3293118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:13:41.3293474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:13:41.3293965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3294484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:13:41.3294772Z ok (1.210s) 2022-12-01T10:13:41.3294862Z 2022-12-01T10:13:41.3295095Z ---------------------------------------------------------------------- 2022-12-01T10:13:41.3295342Z Ran 6 tests in 6.798s 2022-12-01T10:13:41.3295458Z 2022-12-01T10:13:41.3295518Z OK 2022-12-01T10:13:41.3295608Z 2022-12-01T10:13:41.3295680Z Generating XML reports... 2022-12-01T10:13:41.3296283Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221201101334.xml 2022-12-01T10:13:41.3297062Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221201101334.xml 2022-12-01T10:13:41.3297868Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221201101334.xml 2022-12-01T10:13:41.3298251Z 2022-12-01T10:13:41.3298540Z ##[endgroup] 2022-12-01T10:13:41.3299094Z FINISHED PRINTING LOG FILE of distributed/_shard/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_file_system_checkpoint_cpu__za4oypi) 2022-12-01T10:13:41.3299414Z 2022-12-01T10:13:41.3299645Z Running distributed/_shard/checkpoint/test_utils ... [2022-12-01 10:13:41.325973] 2022-12-01T10:13:41.3300180Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:41.326327] 2022-12-01T10:13:42.6963846Z 2022-12-01T10:13:42.6964431Z Expand the folded group to see the log file of distributed/_shard/checkpoint/test_utils 2022-12-01T10:13:42.6965671Z ##[group]PRINTING LOG FILE of distributed/_shard/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_utils_wjyej17q) 2022-12-01T10:13:42.6966341Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_utils 2022-12-01T10:13:42.6966557Z 2022-12-01T10:13:42.6966630Z Running tests... 2022-12-01T10:13:42.6966940Z ---------------------------------------------------------------------- 2022-12-01T10:13:42.6967326Z test_flat_data (__main__.TestMedatadaIndex) ... ok (0.225s) 2022-12-01T10:13:42.6967749Z test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-12-01T10:13:42.6968119Z test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-12-01T10:13:42.6968423Z test_init_convert_offset (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-12-01T10:13:42.6968721Z test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... ok (0.003s) 2022-12-01T10:13:42.6968887Z 2022-12-01T10:13:42.6969081Z ---------------------------------------------------------------------- 2022-12-01T10:13:42.6969327Z Ran 5 tests in 0.230s 2022-12-01T10:13:42.6969440Z 2022-12-01T10:13:42.6969502Z OK 2022-12-01T10:13:42.6969593Z 2022-12-01T10:13:42.6969679Z Generating XML reports... 2022-12-01T10:13:42.6970144Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_utils/TEST-TestMedatadaIndex-20221201101342.xml 2022-12-01T10:13:42.6970414Z 2022-12-01T10:13:42.6970645Z ##[endgroup] 2022-12-01T10:13:42.6971300Z FINISHED PRINTING LOG FILE of distributed/_shard/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_utils_wjyej17q) 2022-12-01T10:13:42.6971565Z 2022-12-01T10:13:42.6971795Z Running distributed/fsdp/test_flatten_params_wrapper ... [2022-12-01 10:13:42.696652] 2022-12-01T10:13:42.6972345Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_flatten_params_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:42.697019] 2022-12-01T10:13:44.1877131Z 2022-12-01T10:13:44.1877661Z Expand the folded group to see the log file of distributed/fsdp/test_flatten_params_wrapper 2022-12-01T10:13:44.1879320Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_flatten_params_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_flatten_params_wrapper_84_58lkf) 2022-12-01T10:13:44.1880041Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper 2022-12-01T10:13:44.1880259Z 2022-12-01T10:13:44.1880335Z Running tests... 2022-12-01T10:13:44.1880682Z ---------------------------------------------------------------------- 2022-12-01T10:13:44.1880978Z test_empty_module (__main__.TestFlattenParams) ... ok (0.222s) 2022-12-01T10:13:44.1881337Z test_flatten_nothing (__main__.TestFlattenParams) ... ok (0.007s) 2022-12-01T10:13:44.1881679Z test_num_params (__main__.TestFlattenParams) ... ok (0.007s) 2022-12-01T10:13:44.1881938Z test_output (__main__.TestFlattenParams) ... ok (0.019s) 2022-12-01T10:13:44.1882272Z test_partial_flattening (__main__.TestFlattenParams) ... ok (0.009s) 2022-12-01T10:13:44.1882718Z test_sharded_flat_param (__main__.TestFlattenParams) ... ok (0.007s) 2022-12-01T10:13:44.1883075Z test_shared_params_num_params (__main__.TestFlattenParams) ... ok (0.007s) 2022-12-01T10:13:44.1883404Z test_shared_params_output (__main__.TestFlattenParams) ... ok (0.016s) 2022-12-01T10:13:44.1883714Z test_shared_params_pnorm_after_step (__main__.TestFlattenParams) ... ok (0.030s) 2022-12-01T10:13:44.1884050Z test_empty_module (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.001s) 2022-12-01T10:13:44.1884385Z test_flatten_nothing (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1884730Z test_num_params (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1885063Z test_output (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1885404Z test_partial_flattening (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.001s) 2022-12-01T10:13:44.1885792Z test_sharded_flat_param (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.003s) 2022-12-01T10:13:44.1886253Z test_shared_params_num_params (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1886622Z test_shared_params_output (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1886988Z test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1887339Z test_empty_module (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1887702Z test_flatten_nothing (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1888056Z test_num_params (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1888390Z test_output (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1888745Z test_partial_flattening (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.001s) 2022-12-01T10:13:44.1889118Z test_sharded_flat_param (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.003s) 2022-12-01T10:13:44.1889489Z test_shared_params_num_params (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1889850Z test_shared_params_output (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1890233Z test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-12-01T10:13:44.1890444Z 2022-12-01T10:13:44.1890669Z ---------------------------------------------------------------------- 2022-12-01T10:13:44.1890913Z Ran 27 tests in 0.339s 2022-12-01T10:13:44.1891013Z 2022-12-01T10:13:44.1891089Z OK (skipped=18) 2022-12-01T10:13:44.1891196Z 2022-12-01T10:13:44.1891280Z Generating XML reports... 2022-12-01T10:13:44.1891765Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20221201101343.xml 2022-12-01T10:13:44.1892440Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20221201101343.xml 2022-12-01T10:13:44.1893086Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20221201101343.xml 2022-12-01T10:13:44.1893379Z 2022-12-01T10:13:44.1893646Z ##[endgroup] 2022-12-01T10:13:44.1894136Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_flatten_params_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_flatten_params_wrapper_84_58lkf) 2022-12-01T10:13:44.1894445Z 2022-12-01T10:13:44.1894653Z Running distributed/rpc/test_tensorpipe_agent ... [2022-12-01 10:13:44.188141] 2022-12-01T10:13:44.1895219Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:44.188477] 2022-12-01T10:40:42.6481044Z 2022-12-01T10:40:42.6481563Z Expand the folded group to see the log file of distributed/rpc/test_tensorpipe_agent 2022-12-01T10:40:42.6482493Z ##[group]PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_elt7x7yf) 2022-12-01T10:40:42.6513399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx8sixr9 2022-12-01T10:40:42.6514078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx8sixr9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6515228Z , <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison_uneven_inputs>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_sparse_grads>]> 2022-12-01T10:40:42.6516297Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) 2022-12-01T10:40:42.6516700Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) 2022-12-01T10:40:42.6517245Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) 2022-12-01T10:40:42.6517722Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) 2022-12-01T10:40:42.6519142Z , <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside_uneven_inputs>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_no_ddp>]> 2022-12-01T10:40:42.6520346Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-12-01T10:40:42.6520920Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-12-01T10:40:42.6521509Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-12-01T10:40:42.6522112Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-12-01T10:40:42.6532148Z , <__main__.TensorPipeDistAutogradTest testMethod=test_autograd_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_autograd_engine_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_complex_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_dtypes>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_tensor_dims>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_invalid_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_output_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_roots>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_round_trips>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_no_grad_on_tensor>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_python_udf_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_multi>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_nested>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_script_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_self>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_send_function>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_verify_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_backwards_nested_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_clean_context_during_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_nested_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_no_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_debug_info>, <__main__.TensorPipeDistAutogradTest testMethod=test_dist_autograd_profiling>, <__main__.TensorPipeDistAutogradTest testMethod=test_error_in_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_copy_sparse_indices_extra_ref>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_mixed_requires_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward_with_errors>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy_sparse>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_post_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_remote_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_rpc_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_thread_local_context_id>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps_torchscript_functions>, <__main__.TensorPipeDistAutogradTest testMethod=test_worker_ids_recorded>]> 2022-12-01T10:40:42.6541598Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6542057Z test_autograd_context (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6543817Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6544301Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6544945Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6545490Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6546030Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6546539Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6547070Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6547627Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6548185Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6548706Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6549224Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6549756Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6550307Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6550809Z test_backward_rref (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6551300Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6551828Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6552305Z test_backward_simple (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6552822Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6553358Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6553901Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6554485Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6555072Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6555610Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6556060Z test_backward_without_context (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6556543Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6557056Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6557560Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6558088Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6558747Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6559280Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6559697Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6560149Z test_debug_info (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6560503Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6560951Z test_error_in_context (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6561352Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6561789Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6562246Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6562728Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6563252Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6563913Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6564434Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6565005Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6565621Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6566086Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6566686Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6567141Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6567605Z test_multiple_backward (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6568166Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6568744Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6569271Z test_nested_context (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6569781Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6570314Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6570902Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6571493Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6572071Z test_post_hooks (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6572535Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6572981Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6573602Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6574125Z test_trainer_ps (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6574648Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6575197Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.6576313Z , <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception_on_constructor>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_none_grads>]> 2022-12-01T10:40:42.6577352Z test_dist_optim (__main__.TensorPipeDistOptimizerTest) 2022-12-01T10:40:42.6577866Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) 2022-12-01T10:40:42.6578406Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) 2022-12-01T10:40:42.6578983Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) 2022-12-01T10:40:42.6580110Z , <__main__.TensorPipeJitDistAutogradTest testMethod=test_get_gradients>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_jit_fork_within_context>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_restore_context_after_swtich_to_jit_thread>]> 2022-12-01T10:40:42.6581252Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest) 2022-12-01T10:40:42.6581705Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest) 2022-12-01T10:40:42.6582014Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) 2022-12-01T10:40:42.6582385Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) 2022-12-01T10:40:42.6590110Z , <__main__.TensorPipeJitRpcTest testMethod=test_all_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_args_and_kwargs_contain_different_types>, <__main__.TensorPipeJitRpcTest testMethod=test_args_kwargs_are_neither_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote_multi>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_decorator_order>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_call_fork_in_jit_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_python_function_remotely_from_script_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_call_rpc_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_not_exists_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_raises_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_with_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_class_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_module_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_script_module_on_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_future_passed_between_python_and_jit>, <__main__.TensorPipeJitRpcTest testMethod=test_future_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_kwargs_not_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_less_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_load_script_module_with_pickled_rref>, <__main__.TensorPipeJitRpcTest testMethod=test_local_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_more_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_my_script_module_with_rrefs>, <__main__.TensorPipeJitRpcTest testMethod=test_no_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_jit_end_callbacks_with_fork>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_on_caller_rpc_async>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_module>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_class_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_module_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_async_jit_profiled>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_torchscript_record_function>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_as_arg_and_return>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_is_owner>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_jit_pickle_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_list_mutate>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_some_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_functions_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_unexepected_kwarg_is_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed_remote>]> 2022-12-01T10:40:42.6598049Z test_add_done_callback (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6598663Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6599218Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6599741Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6600174Z test_async_function_remote (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6600649Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6601126Z test_async_function_simple (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6601741Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6602273Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6602689Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6603082Z test_async_script_throw (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6603438Z test_async_script_udf (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6603866Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6604370Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6604866Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6605397Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6605949Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6606371Z test_callback_chain (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6606769Z test_callback_simple (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6607158Z test_callback_with_exception (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6607582Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6608135Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6608606Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6609139Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6609548Z test_future_python_annotation (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6609976Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6610458Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6610958Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6611410Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6611883Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6612405Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6612903Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6613467Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6613961Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6614414Z test_remote_script_module (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6614825Z test_remote_script_throw (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6615299Z test_remote_script_udf (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6615849Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6616415Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6616905Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6617417Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6617880Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6618282Z test_rref_is_owner (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6618794Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6619210Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6619584Z test_rref_local_value (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6620019Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6620500Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6620947Z test_torchscript_function (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6621410Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6621898Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6622378Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6622809Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6623179Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) 2022-12-01T10:40:42.6623748Z ]> 2022-12-01T10:40:42.6624280Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) 2022-12-01T10:40:42.6624855Z ]> 2022-12-01T10:40:42.6625379Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) 2022-12-01T10:40:42.6627788Z , <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_with_kwargs>, <__main__.TensorPipeRemoteModuleTest testMethod=test_get_module_rref>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_parameters>, <__main__.TensorPipeRemoteModuleTest testMethod=test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire>, <__main__.TensorPipeRemoteModuleTest testMethod=test_train_eval>, <__main__.TensorPipeRemoteModuleTest testMethod=test_unsupported_methods>]> 2022-12-01T10:40:42.6630088Z test_bad_module (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6630579Z test_forward_async (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6631098Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6631511Z test_forward_sync (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6631923Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6632373Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6632796Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6633273Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6633805Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6634298Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6634816Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6635327Z test_train_eval (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6635737Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) 2022-12-01T10:40:42.6663916Z , <__main__.TensorPipeRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeRpcTest testMethod=test_add_with_id>, <__main__.TensorPipeRpcTest testMethod=test_all_gather>, <__main__.TensorPipeRpcTest testMethod=test_all_gather_timeout>, <__main__.TensorPipeRpcTest testMethod=test_async_add>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_async>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_cbs_jit_call>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks_new_signatures>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_build_rpc_profiling_key>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_self>, <__main__.TensorPipeRpcTest testMethod=test_call_method_on_rref>, <__main__.TensorPipeRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeRpcTest testMethod=test_callback_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_callback_multi>, <__main__.TensorPipeRpcTest testMethod=test_callback_none>, <__main__.TensorPipeRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_error>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_ret>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_num>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_type>, <__main__.TensorPipeRpcTest testMethod=test_cannot_infer_backend_from_options>, <__main__.TensorPipeRpcTest testMethod=test_deadlock>, <__main__.TensorPipeRpcTest testMethod=test_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_default_timeout_used>, <__main__.TensorPipeRpcTest testMethod=test_disable_gil_profiling>, <__main__.TensorPipeRpcTest testMethod=test_dist_init_decorator>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name_2>, <__main__.TensorPipeRpcTest testMethod=test_expected_src>, <__main__.TensorPipeRpcTest testMethod=test_function_not_on_callee>, <__main__.TensorPipeRpcTest testMethod=test_future_done>, <__main__.TensorPipeRpcTest testMethod=test_future_done_exception>, <__main__.TensorPipeRpcTest testMethod=test_future_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_future_nested_callback>, <__main__.TensorPipeRpcTest testMethod=test_future_wait_twice>, <__main__.TensorPipeRpcTest testMethod=test_get_worker_infos>, <__main__.TensorPipeRpcTest testMethod=test_graceful_shutdown_with_uneven_workload>, <__main__.TensorPipeRpcTest testMethod=test_handle_send_exceptions>, <__main__.TensorPipeRpcTest testMethod=test_ignore_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_init_pg_then_rpc>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_then_pg>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_twice>, <__main__.TensorPipeRpcTest testMethod=test_int_callee>, <__main__.TensorPipeRpcTest testMethod=test_invalid_names>, <__main__.TensorPipeRpcTest testMethod=test_local_rref_no_fork>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown_with_rpc>, <__main__.TensorPipeRpcTest testMethod=test_local_value_not_on_owner>, <__main__.TensorPipeRpcTest testMethod=test_mark_future_twice>, <__main__.TensorPipeRpcTest testMethod=test_multi_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_multi_layer_nested_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_multi_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_multi_rpc>, <__main__.TensorPipeRpcTest testMethod=test_my_parameter_server>, <__main__.TensorPipeRpcTest testMethod=test_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_nested_rpc>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref_stress>, <__main__.TensorPipeRpcTest testMethod=test_non_cont_tensors>, <__main__.TensorPipeRpcTest testMethod=test_non_garbage_collected_user_rref_due_to_local_circular_dependency>, <__main__.TensorPipeRpcTest testMethod=test_nonzero>, <__main__.TensorPipeRpcTest testMethod=test_owner_equality>, <__main__.TensorPipeRpcTest testMethod=test_owner_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_pass_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_pg_init_no_rpc_init>, <__main__.TensorPipeRpcTest testMethod=test_pickle_future>, <__main__.TensorPipeRpcTest testMethod=test_profiler_export_trace>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_key_names>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_memory>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_record_shapes>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_py_built_in>, <__main__.TensorPipeRpcTest testMethod=test_py_class_constructor>, <__main__.TensorPipeRpcTest testMethod=test_py_class_instance_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_static_method>, <__main__.TensorPipeRpcTest testMethod=test_py_function_exception>, <__main__.TensorPipeRpcTest testMethod=test_py_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_nested_pickle>, <__main__.TensorPipeRpcTest testMethod=test_py_no_return_result>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func_escaped_str>, <__main__.TensorPipeRpcTest testMethod=test_py_rpc_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args_user_share>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_in_container>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_py_user_defined>, <__main__.TensorPipeRpcTest testMethod=test_register_rpc_backend_and_set_and_start_rpc_backend>, <__main__.TensorPipeRpcTest testMethod=test_reinit>, <__main__.TensorPipeRpcTest testMethod=test_remote_same_worker>, <__main__.TensorPipeRpcTest testMethod=test_remote_throw>, <__main__.TensorPipeRpcTest testMethod=test_remote_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_return_future>, <__main__.TensorPipeRpcTest testMethod=test_return_future_async>, <__main__.TensorPipeRpcTest testMethod=test_return_future_remote>, <__main__.TensorPipeRpcTest testMethod=test_return_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_all>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_multithreaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_partial_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_remote_record_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_return_rref>, <__main__.TensorPipeRpcTest testMethod=test_rpc_timeouts>, <__main__.TensorPipeRpcTest testMethod=test_rref_context_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_rref_forward_chain>, <__main__.TensorPipeRpcTest testMethod=test_rref_get_future>, <__main__.TensorPipeRpcTest testMethod=test_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_non_exist>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_reuse>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_py_pickle_not_supported>, <__main__.TensorPipeRpcTest testMethod=test_rref_str>, <__main__.TensorPipeRpcTest testMethod=test_rref_timeout>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_slow_init>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_scalar_add>, <__main__.TensorPipeRpcTest testMethod=test_self_add>, <__main__.TensorPipeRpcTest testMethod=test_self_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_send_to_rank>, <__main__.TensorPipeRpcTest testMethod=test_server_process_global_profiler>, <__main__.TensorPipeRpcTest testMethod=test_set_and_get_default_rpc_timeout>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_errors>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_followed_by_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc_torchscript>, <__main__.TensorPipeRpcTest testMethod=test_stress_light_rpc>, <__main__.TensorPipeRpcTest testMethod=test_use_rpc_pickler>, <__main__.TensorPipeRpcTest testMethod=test_use_rref_after_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_user_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed_remote>, <__main__.TensorPipeRpcTest testMethod=test_wait_all>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_builtin>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_python>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_script_function>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_multiple_call>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_body>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_partial_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_dense>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_twice_dense>, <__main__.TensorPipeRpcTest testMethod=test_worker_id>, <__main__.TensorPipeRpcTest testMethod=test_worker_info_pickle>, <__main__.TensorPipeRpcTest testMethod=test_world_size_one>, <__main__.TensorPipeRpcTest testMethod=test_wrong_types>]> 2022-12-01T10:40:42.6689508Z test_add (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6689927Z test_add_done_callback (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6690268Z test_add_with_id (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6690623Z test_all_gather (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6691020Z test_all_gather_timeout (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6691413Z test_async_add (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6691819Z test_async_class_method (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6692217Z test_async_class_method_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6692662Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6693127Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6693600Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6694065Z test_async_function_chained (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6694526Z test_async_function_chained_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6695001Z test_async_function_multi_chained (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6695473Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6695972Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6696428Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6696881Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6697372Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6697849Z test_async_function_nested (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6698458Z test_async_function_nested_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6698864Z test_async_function_raise (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6699267Z test_async_function_raise_async (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6699705Z test_async_function_raise_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6700148Z test_async_function_simple (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6700521Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6701055Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6701470Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6701936Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6702461Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6702940Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6703453Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6703995Z test_async_record_function_double_end_callbacks_new_signatures (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6704530Z test_async_static_method (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6704951Z test_async_static_method_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6705411Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6705873Z test_builtin_remote_ret (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6706290Z test_builtin_remote_self (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6706717Z test_call_method_on_rref (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6707161Z test_callback_chain (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6707581Z test_callback_in_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6708017Z test_callback_multi (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6708426Z test_callback_none (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6708865Z test_callback_simple (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6709302Z test_callback_with_error (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6709746Z test_callback_with_ret (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6710189Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6710646Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6711117Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6711553Z test_deadlock (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6711932Z test_debug_info (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6712347Z test_default_timeout_used (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6712798Z test_disable_gil_profiling (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6713237Z test_dist_init_decorator (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6713631Z test_duplicate_name (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6713994Z test_duplicate_name_2 (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6714401Z test_expected_src (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6714794Z test_function_not_on_callee (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6715217Z test_future_done (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6715631Z test_future_done_exception (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6716034Z test_future_in_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6716463Z test_future_nested_callback (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6716851Z test_future_wait_twice (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6717187Z test_get_worker_infos (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6717591Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6718015Z test_handle_send_exceptions (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6718483Z test_ignore_rref_leak (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6718972Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6719390Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6719785Z test_init_rpc_twice (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6720185Z test_int_callee (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6720600Z test_invalid_names (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6721018Z test_local_rref_no_fork (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6721420Z test_local_shutdown (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6721921Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6722361Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6722760Z test_mark_future_twice (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6723187Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6723646Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6724115Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6724546Z test_multi_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6724966Z test_my_parameter_server (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6725434Z test_nested_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6725826Z test_nested_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6778201Z test_nested_rref (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6778634Z test_nested_rref_stress (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6779038Z test_non_cont_tensors (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6779481Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6779874Z test_nonzero (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6780222Z test_owner_equality (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6780613Z test_owner_rref_backward (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6780984Z test_pass_local_rrefs (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6781353Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6781724Z test_pickle_future (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6782108Z test_profiler_export_trace (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6782480Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6782921Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6783262Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6783624Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6784018Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6784422Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6784883Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6785286Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6785692Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6786076Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6786521Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6786964Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6787361Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6787808Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6788211Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6788647Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6789088Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6789524Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6790119Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6790581Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6791019Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6791450Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6791879Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6792367Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6792767Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6793155Z test_py_built_in (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6793513Z test_py_class_constructor (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6793898Z test_py_class_instance_method (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6794263Z test_py_class_method (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6794589Z test_py_class_static_method (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6794995Z test_py_function_exception (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6795364Z test_py_multi_async_call (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6795768Z test_py_nested_pickle (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6796134Z test_py_no_return_result (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6796514Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6796933Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6797357Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6797689Z test_py_rref_args (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6798072Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6798562Z test_py_tensors (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6798948Z test_py_tensors_in_container (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6799322Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6799705Z test_py_udf_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6800096Z test_py_user_defined (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6800563Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6800968Z test_reinit (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6801325Z test_remote_same_worker (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6801693Z test_remote_throw (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6802066Z test_remote_with_exception (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6802434Z test_return_future (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6802797Z test_return_future_async (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6803095Z test_return_future_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6803445Z test_return_local_rrefs (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6803816Z test_rpc_barrier_all (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6804219Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6804589Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6804996Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6805411Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6805851Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6806311Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6806708Z test_rpc_return_rref (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6807057Z test_rpc_timeouts (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6807429Z test_rref_context_debug_info (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6807894Z test_rref_forward_chain (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6808258Z test_rref_get_future (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6808601Z test_rref_leak (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6809089Z test_rref_proxy_class (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6809491Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6809836Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6810219Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6810582Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6810923Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6811365Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6811754Z test_rref_str (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6812099Z test_rref_timeout (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6812467Z test_rref_type_blocking (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6812827Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6813202Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6813602Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6814038Z test_rref_type_slow_init (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6814481Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6814945Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6815386Z test_scalar_add (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6815803Z test_self_add (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6816220Z test_self_py_udf_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6816669Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6817139Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6817618Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6818077Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6818544Z test_send_to_rank (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6819023Z test_server_process_global_profiler (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6819504Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6819950Z test_shutdown_errors (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6820373Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6820787Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6821183Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6821564Z test_stress_light_rpc (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6821959Z test_use_rpc_pickler (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6822349Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6822724Z test_user_rref_backward (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6823108Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6823473Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6823818Z test_wait_all (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6824150Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6824566Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6824971Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6825380Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6825751Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6826084Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6826462Z test_wait_all_timeout (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6826804Z test_wait_all_with_exception (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6827075Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6827361Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6827778Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6828139Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6828458Z test_worker_id (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6828715Z test_worker_info_pickle (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6828975Z test_world_size_one (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6829213Z test_wrong_types (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.6833738Z , <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_multiple_round_trips_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_no_grad_on_tensor_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_multi_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_nested_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_script_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_self_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backwards_nested_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_embedding_bag_with_no_grad_tensors>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_mixed_requires_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_multiple_backward_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_nested_backward_accumulate_grads_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_remote_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_rpc_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_trainer_ps_sparse>]> 2022-12-01T10:40:42.6838581Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6839008Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6839422Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6839875Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6840272Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6840750Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6841152Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6841549Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6841998Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6842386Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6842772Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6843180Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6843594Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6844011Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6844410Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6844813Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6845219Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6845632Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6846022Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6846434Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6846855Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6847245Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6847641Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6848106Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6848503Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6848901Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6849383Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6849823Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6850223Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6850613Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6850988Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-12-01T10:40:42.6855687Z , <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_self_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_and_static_init_rpc_together>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc_without_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_new_rank_can_communicated_with_existing_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_infer_backend_from_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_mismatched_type_for_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_my_parameter_server_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_stress_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_op_with_invalid_args>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rpc_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_user_share_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_sparse_tensors_in_container>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_non_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_proxy_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_send_to_rank_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_set_and_get_num_worker_threads>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_stress_heavy_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_options_throw_on_timedelta_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_set_default_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_twice_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_world_size_one_sparse>]> 2022-12-01T10:40:42.6860293Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6860647Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6861011Z test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6861391Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6861807Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6862183Z test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6862539Z test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6862914Z test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6863289Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6863639Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6863982Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6864353Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6864683Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6865015Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6865341Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6865665Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6866024Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6866344Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6866678Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6867010Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6867331Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6867676Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6868025Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6868379Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6868721Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6869064Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6869398Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6869740Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6870103Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6870467Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6870837Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6871169Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6871504Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6871841Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6872196Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6872563Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6872904Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6873248Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6873576Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-12-01T10:40:42.6874269Z , <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire_script_not_supported>]> 2022-12-01T10:40:42.6874970Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-12-01T10:40:42.6875356Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-12-01T10:40:42.6875749Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-12-01T10:40:42.6876376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6876716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6877184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6877519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6877866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeq7on1uu 2022-12-01T10:40:42.6878423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeq7on1uu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6878900Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.6879176Z 2022-12-01T10:40:42.6879249Z Running tests... 2022-12-01T10:40:42.6879559Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6880429Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69662 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.218s) 2022-12-01T10:40:42.6880888Z 2022-12-01T10:40:42.6881082Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6881312Z Ran 1 test in 0.218s 2022-12-01T10:40:42.6881423Z 2022-12-01T10:40:42.6881497Z OK (skipped=1) 2022-12-01T10:40:42.6881603Z 2022-12-01T10:40:42.6881688Z Generating XML reports... 2022-12-01T10:40:42.6882168Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221201101346.xml 2022-12-01T10:40:42.6882736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6883068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6883493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6883825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6884173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlly5sq4 2022-12-01T10:40:42.6884577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlly5sq4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6885037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.6885247Z 2022-12-01T10:40:42.6885318Z Running tests... 2022-12-01T10:40:42.6885624Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6886500Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72891 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.218s) 2022-12-01T10:40:42.6886966Z 2022-12-01T10:40:42.6887163Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6887392Z Ran 1 test in 0.218s 2022-12-01T10:40:42.6887505Z 2022-12-01T10:40:42.6887578Z OK (skipped=1) 2022-12-01T10:40:42.6887737Z 2022-12-01T10:40:42.6887824Z Generating XML reports... 2022-12-01T10:40:42.6888302Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221201101348.xml 2022-12-01T10:40:42.6888869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6889202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6889632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6889966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6890378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5_ohor0 2022-12-01T10:40:42.6890781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5_ohor0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6891237Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.6891447Z 2022-12-01T10:40:42.6891519Z Running tests... 2022-12-01T10:40:42.6891825Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6892278Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10899 2022-12-01T10:40:42.6892686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10900 2022-12-01T10:40:42.6893014Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10901 2022-12-01T10:40:42.6893342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10902 2022-12-01T10:40:42.6893796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6894113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6894540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6894891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6895232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5s0r168 2022-12-01T10:40:42.6895632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5s0r168/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6896112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6896442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6896860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6897204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6897547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1cf2lx5 2022-12-01T10:40:42.6897933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1cf2lx5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6898420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6898747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6899175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6899505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6899849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn90x91d 2022-12-01T10:40:42.6900247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn90x91d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6900731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6901047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6901472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6901818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6902148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjzcria9 2022-12-01T10:40:42.6902546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjzcria9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6902923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.6903304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.6903638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.6903979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.6904340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.6904722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.6905089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.6905583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.6905977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.6906456Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.6906976Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.6907490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.6907876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6908221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6908572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6908921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6909275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6909616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6909977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6910324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6910564Z ok (2.238s) 2022-12-01T10:40:42.6910668Z 2022-12-01T10:40:42.6910869Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6911109Z Ran 1 test in 2.238s 2022-12-01T10:40:42.6911220Z 2022-12-01T10:40:42.6911279Z OK 2022-12-01T10:40:42.6911356Z 2022-12-01T10:40:42.6911442Z Generating XML reports... 2022-12-01T10:40:42.6911933Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221201101349.xml 2022-12-01T10:40:42.6912501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6912823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6913252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6913599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6913945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxpbx6ny 2022-12-01T10:40:42.6914337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxpbx6ny/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6914805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.6915012Z 2022-12-01T10:40:42.6915085Z Running tests... 2022-12-01T10:40:42.6915373Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6915827Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11154 2022-12-01T10:40:42.6916249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11155 2022-12-01T10:40:42.6916585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11156 2022-12-01T10:40:42.6916902Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11157 2022-12-01T10:40:42.6917349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6917707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6918137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6918602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6918951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4bn2scrb 2022-12-01T10:40:42.6919355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4bn2scrb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6919830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6920162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6920590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6920940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6921273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5sfqx52 2022-12-01T10:40:42.6921673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5sfqx52/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6922159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6922479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6922905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6923246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6923592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolhc3kpw 2022-12-01T10:40:42.6923981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolhc3kpw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6924465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6924792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6925213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6925545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6925884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwksutl9m 2022-12-01T10:40:42.6926280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwksutl9m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6926644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.6927090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.6927439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.6927889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.6928239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.6928682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.6929058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.6929413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.6929918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.6930443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.6931013Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.6931511Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.6931895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6932261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6932619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6932962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6933219Z ok (2.238s) 2022-12-01T10:40:42.6933323Z 2022-12-01T10:40:42.6933521Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6933752Z Ran 1 test in 2.238s 2022-12-01T10:40:42.6933877Z 2022-12-01T10:40:42.6933938Z OK 2022-12-01T10:40:42.6934029Z 2022-12-01T10:40:42.6934113Z Generating XML reports... 2022-12-01T10:40:42.6934604Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221201101353.xml 2022-12-01T10:40:42.6935158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6935490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6935910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6936240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6936577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9t1yl0wk 2022-12-01T10:40:42.6936978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9t1yl0wk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6937435Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.6937638Z 2022-12-01T10:40:42.6937698Z Running tests... 2022-12-01T10:40:42.6937994Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.6938415Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11421 2022-12-01T10:40:42.6938838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11422 2022-12-01T10:40:42.6939155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11423 2022-12-01T10:40:42.6939478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11424 2022-12-01T10:40:42.6939802Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11425 2022-12-01T10:40:42.6940117Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11426 2022-12-01T10:40:42.6940562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6940890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6941307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6941667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6941999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6l0lu6yf 2022-12-01T10:40:42.6942388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6l0lu6yf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6942856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6943201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6943621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6943962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6944283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v1_yu8x 2022-12-01T10:40:42.6944680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v1_yu8x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6945050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.6945499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6945812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6946234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6946573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6946901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpft2g9mm8 2022-12-01T10:40:42.6947286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpft2g9mm8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6947769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6948083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6948491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6948830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6949160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunzqws14 2022-12-01T10:40:42.6949542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunzqws14/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6950016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6950333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6950751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6951078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6951409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_p1jjaww 2022-12-01T10:40:42.6951793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_p1jjaww/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6952157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-12-01T10:40:42.6952599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.6952915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.6953334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.6953660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.6954024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprf12nc39 2022-12-01T10:40:42.6954415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprf12nc39/_remote_module_non_scriptable.py 2022-12-01T10:40:42.6954779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.6955110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.6955449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.6955818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-12-01T10:40:42.6956259Z 2022-12-01 10:13:59,451 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-12-01T10:40:42.6956756Z 2022-12-01 10:13:59,451 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.6957230Z 2022-12-01 10:13:59,461 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-12-01T10:40:42.6957730Z 2022-12-01 10:13:59,461 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.6958332Z 2022-12-01 10:13:59,481 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-12-01T10:40:42.6958764Z 2022-12-01 10:13:59,481 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-12-01T10:40:42.6959259Z 2022-12-01 10:13:59,481 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.6959781Z 2022-12-01 10:13:59,481 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.6960254Z 2022-12-01 10:13:59,491 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-12-01T10:40:42.6960690Z 2022-12-01 10:13:59,491 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-12-01T10:40:42.6961050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.6961416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.6961773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.6962127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.6962483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-12-01T10:40:42.6962837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-12-01T10:40:42.6963313Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.6963825Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.6964335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.6964844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.6965338Z 2022-12-01 10:13:59,553 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-12-01T10:40:42.6965834Z 2022-12-01 10:13:59,553 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-12-01T10:40:42.6966378Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.6966877Z 2022-12-01 10:13:59,553 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-12-01T10:40:42.6967354Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.6967910Z 2022-12-01 10:13:59,554 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-12-01T10:40:42.6968328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-12-01T10:40:42.6968786Z 2022-12-01 10:13:59,554 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-12-01T10:40:42.6969132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-12-01T10:40:42.6969574Z 2022-12-01 10:13:59,575 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-12-01T10:40:42.6970007Z 2022-12-01 10:13:59,576 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-12-01T10:40:42.6970362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-12-01T10:40:42.6970713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-12-01T10:40:42.6971080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-12-01T10:40:42.6971445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-12-01T10:40:42.6971920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.6972430Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.6972936Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.6973438Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.6973926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.6974428Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.6974907Z 2022-12-01 10:13:59,728 ddp_under_dist_autograd_test.py:153 INFO p:process 0 t:Dummy-2: Use DDP for the second local net. 2022-12-01T10:40:42.6975350Z 2022-12-01 10:13:59,728 ddp_under_dist_autograd_test.py:153 INFO p:process 3 t:Dummy-2: Use DDP for the second local net. 2022-12-01T10:40:42.6975787Z 2022-12-01 10:13:59,734 ddp_under_dist_autograd_test.py:153 INFO p:process 1 t:Dummy-2: Use DDP for the second local net. 2022-12-01T10:40:42.6976229Z 2022-12-01 10:13:59,740 ddp_under_dist_autograd_test.py:153 INFO p:process 2 t:Dummy-2: Use DDP for the second local net. 2022-12-01T10:40:42.6976815Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.6977247Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.6977673Z 2022-12-01 10:13:59,773 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.6978211Z 2022-12-01 10:13:59,773 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-12-01T10:40:42.6978901Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.6979324Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.6979745Z 2022-12-01 10:13:59,774 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.6980280Z 2022-12-01 10:13:59,774 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-12-01T10:40:42.6980942Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.6981362Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.6981781Z 2022-12-01 10:13:59,774 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.6982320Z 2022-12-01 10:13:59,774 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-12-01T10:40:42.6982958Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.6983379Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.6983808Z 2022-12-01 10:13:59,774 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.6984333Z 2022-12-01 10:13:59,774 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-12-01T10:40:42.6984889Z 2022-12-01 10:13:59,830 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.6985200Z [ 1., -1.], 2022-12-01T10:40:42.6985351Z [ 1., 1.], 2022-12-01T10:40:42.6985730Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.6986145Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.6986407Z tensor([[-1., 1.], 2022-12-01T10:40:42.6986663Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-12-01T10:40:42.6986894Z [ 0., -16.]])} 2022-12-01T10:40:42.6987320Z 2022-12-01 10:13:59,830 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.6987623Z [-1., -1.], 2022-12-01T10:40:42.6987807Z [-1., 1.], 2022-12-01T10:40:42.6988188Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.6988598Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.6988842Z tensor([[-1., 1.], 2022-12-01T10:40:42.6989102Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-12-01T10:40:42.6989324Z [ 16., -16.]])} 2022-12-01T10:40:42.6989730Z 2022-12-01 10:13:59,830 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.6990042Z [-1., 1.], 2022-12-01T10:40:42.6990224Z [-1., -1.], 2022-12-01T10:40:42.6990587Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.6990989Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.6991246Z tensor([[-1., 1.], 2022-12-01T10:40:42.6991539Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-12-01T10:40:42.6991761Z [-16., 16.]])} 2022-12-01T10:40:42.6992177Z 2022-12-01 10:13:59,833 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.6992461Z [ 1., 1.], 2022-12-01T10:40:42.6992636Z [ 1., -1.], 2022-12-01T10:40:42.6993011Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.6993444Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.6993698Z tensor([[-1., 1.], 2022-12-01T10:40:42.6993888Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-12-01T10:40:42.6994082Z [ 0., 16.]])} 2022-12-01T10:40:42.6994346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6994693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6995039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6995383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.6995877Z 2022-12-01 10:13:59,851 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.6996186Z [-1., 1.], 2022-12-01T10:40:42.6996370Z [-1., -1.], 2022-12-01T10:40:42.6996740Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.6997130Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.6997387Z tensor([[-1., 1.], 2022-12-01T10:40:42.6997650Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-12-01T10:40:42.6997867Z [-16., 16.]])} 2022-12-01T10:40:42.6998418Z 2022-12-01 10:13:59,852 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.6998744Z [-1., -1.], 2022-12-01T10:40:42.6998936Z [-1., 1.], 2022-12-01T10:40:42.6999305Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.6999715Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.6999972Z tensor([[-1., 1.], 2022-12-01T10:40:42.7000224Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-12-01T10:40:42.7000454Z [ 16., -16.]])} 2022-12-01T10:40:42.7000879Z 2022-12-01 10:13:59,852 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7001160Z [ 1., 1.], 2022-12-01T10:40:42.7001336Z [ 1., -1.], 2022-12-01T10:40:42.7001698Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7002094Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7002340Z tensor([[-1., 1.], 2022-12-01T10:40:42.7002546Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-12-01T10:40:42.7002743Z [ 0., 16.]])} 2022-12-01T10:40:42.7003148Z 2022-12-01 10:13:59,852 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7003458Z [ 1., -1.], 2022-12-01T10:40:42.7003617Z [ 1., 1.], 2022-12-01T10:40:42.7004048Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7004443Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7004700Z tensor([[-1., 1.], 2022-12-01T10:40:42.7004963Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-12-01T10:40:42.7005184Z [ 0., -16.]])} 2022-12-01T10:40:42.7005600Z 2022-12-01 10:13:59,873 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7005915Z [ 1., 1.], 2022-12-01T10:40:42.7006090Z [ 1., -1.], 2022-12-01T10:40:42.7006464Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7006864Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7007114Z tensor([[-1., 1.], 2022-12-01T10:40:42.7007305Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-12-01T10:40:42.7007501Z [ 0., 16.]])} 2022-12-01T10:40:42.7007982Z 2022-12-01 10:13:59,873 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7008291Z [-1., -1.], 2022-12-01T10:40:42.7008479Z [-1., 1.], 2022-12-01T10:40:42.7008865Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7009272Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7009515Z tensor([[-1., 1.], 2022-12-01T10:40:42.7009780Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-12-01T10:40:42.7010007Z [ 16., -16.]])} 2022-12-01T10:40:42.7010412Z 2022-12-01 10:13:59,873 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7010724Z [ 1., -1.], 2022-12-01T10:40:42.7010880Z [ 1., 1.], 2022-12-01T10:40:42.7011250Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7011656Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7011915Z tensor([[-1., 1.], 2022-12-01T10:40:42.7012177Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-12-01T10:40:42.7012392Z [ 0., -16.]])} 2022-12-01T10:40:42.7012801Z 2022-12-01 10:13:59,875 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7013111Z [-1., 1.], 2022-12-01T10:40:42.7013283Z [-1., -1.], 2022-12-01T10:40:42.7013662Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7014057Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7014313Z tensor([[-1., 1.], 2022-12-01T10:40:42.7014563Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-12-01T10:40:42.7014784Z [-16., 16.]])} 2022-12-01T10:40:42.7015142Z 2022-12-01 10:13:59,879 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-12-01T10:40:42.7015578Z 2022-12-01 10:13:59,880 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-12-01T10:40:42.7016013Z 2022-12-01 10:13:59,880 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-12-01T10:40:42.7016451Z 2022-12-01 10:13:59,881 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-12-01T10:40:42.7016918Z 2022-12-01 10:13:59,881 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-12-01T10:40:42.7017159Z ok (3.347s) 2022-12-01T10:40:42.7017257Z 2022-12-01T10:40:42.7017455Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7017690Z Ran 1 test in 3.347s 2022-12-01T10:40:42.7017796Z 2022-12-01T10:40:42.7017844Z OK 2022-12-01T10:40:42.7017931Z 2022-12-01T10:40:42.7018053Z Generating XML reports... 2022-12-01T10:40:42.7018566Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221201101357.xml 2022-12-01T10:40:42.7019151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7019466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7019884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7020218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7020547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply_5y52p 2022-12-01T10:40:42.7020943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply_5y52p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7021412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7021621Z 2022-12-01T10:40:42.7021693Z Running tests... 2022-12-01T10:40:42.7021977Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7022396Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11835 2022-12-01T10:40:42.7022813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11836 2022-12-01T10:40:42.7023131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11837 2022-12-01T10:40:42.7023456Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11838 2022-12-01T10:40:42.7023772Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11839 2022-12-01T10:40:42.7024089Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11840 2022-12-01T10:40:42.7024525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7024849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7025270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7025613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7025937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ufduzfz 2022-12-01T10:40:42.7026337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ufduzfz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7026815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7027124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7027543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7027885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7028217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4e0vod7 2022-12-01T10:40:42.7028601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4e0vod7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7029078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7029428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7029840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7030177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7030513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3txur4pv 2022-12-01T10:40:42.7030938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3txur4pv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7031304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-12-01T10:40:42.7031759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7032081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7032502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7032826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7033165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uydrdpn 2022-12-01T10:40:42.7033559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uydrdpn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7033923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7034377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7034694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7035117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7035446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7035784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8_1ul1x 2022-12-01T10:40:42.7036181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8_1ul1x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7036650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7036972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7037398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7037735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7038064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_obksec 2022-12-01T10:40:42.7038580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_obksec/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7038956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-12-01T10:40:42.7039296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7039627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7039963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7040416Z 2022-12-01 10:14:04,193 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-12-01T10:40:42.7040915Z 2022-12-01 10:14:04,193 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7041392Z 2022-12-01 10:14:04,213 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-12-01T10:40:42.7041945Z 2022-12-01 10:14:04,213 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7042420Z 2022-12-01 10:14:04,223 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-12-01T10:40:42.7042912Z 2022-12-01 10:14:04,224 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7043388Z 2022-12-01 10:14:04,233 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-12-01T10:40:42.7043880Z 2022-12-01 10:14:04,233 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-12-01T10:40:42.7044388Z 2022-12-01 10:14:04,234 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7044875Z 2022-12-01 10:14:04,234 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-12-01T10:40:42.7045227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7045589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7045956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7046305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7046668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-12-01T10:40:42.7047032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-12-01T10:40:42.7047523Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7048092Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7048603Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7049113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7049615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7050116Z 2022-12-01 10:14:04,297 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-12-01T10:40:42.7050604Z 2022-12-01 10:14:04,297 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-12-01T10:40:42.7051104Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7051612Z 2022-12-01 10:14:04,297 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-12-01T10:40:42.7052057Z 2022-12-01 10:14:04,298 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-12-01T10:40:42.7052422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-12-01T10:40:42.7052793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-12-01T10:40:42.7053275Z 2022-12-01 10:14:04,307 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-12-01T10:40:42.7053724Z 2022-12-01 10:14:04,328 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-2: Initing RemoteNet with 5 3 2022-12-01T10:40:42.7054229Z 2022-12-01 10:14:04,329 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-3: Initing RemoteEM with 2 3 2022-12-01T10:40:42.7054584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-12-01T10:40:42.7054948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-12-01T10:40:42.7055296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-12-01T10:40:42.7055779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7056197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-12-01T10:40:42.7056664Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7057177Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7057684Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7058192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7058673Z 2022-12-01 10:14:04,461 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7059155Z 2022-12-01 10:14:04,461 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-12-01T10:40:42.7059654Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7060145Z 2022-12-01 10:14:04,467 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7060607Z 2022-12-01 10:14:04,467 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-12-01T10:40:42.7061075Z 2022-12-01 10:14:04,468 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7061542Z 2022-12-01 10:14:04,468 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7062010Z 2022-12-01 10:14:04,468 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-12-01T10:40:42.7062472Z 2022-12-01 10:14:04,468 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-12-01T10:40:42.7063070Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.7063499Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.7064012Z 2022-12-01 10:14:04,494 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-12-01T10:40:42.7064649Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.7065071Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.7065582Z 2022-12-01 10:14:04,494 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-12-01T10:40:42.7066227Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.7066654Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.7067202Z 2022-12-01 10:14:04,494 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-12-01T10:40:42.7067849Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.7068276Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.7068814Z 2022-12-01 10:14:04,497 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-12-01T10:40:42.7069369Z 2022-12-01 10:14:04,541 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7069692Z [-1., -1.], 2022-12-01T10:40:42.7069884Z [-1., 1.], 2022-12-01T10:40:42.7070266Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7070681Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7070936Z tensor([[-1., 1.], 2022-12-01T10:40:42.7071142Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7071325Z [0., 0.]])} 2022-12-01T10:40:42.7071741Z 2022-12-01 10:14:04,543 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7072048Z [-1., 1.], 2022-12-01T10:40:42.7072222Z [-1., -1.], 2022-12-01T10:40:42.7072599Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7073006Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7073263Z tensor([[-1., 1.], 2022-12-01T10:40:42.7073458Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7073653Z [0., 0.]])} 2022-12-01T10:40:42.7074066Z 2022-12-01 10:14:04,543 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7074367Z [ 1., -1.], 2022-12-01T10:40:42.7074521Z [ 1., 1.], 2022-12-01T10:40:42.7074905Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7075303Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7075559Z tensor([[-1., 1.], 2022-12-01T10:40:42.7075763Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7075956Z [0., 0.]])} 2022-12-01T10:40:42.7076358Z 2022-12-01 10:14:04,543 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7076639Z [ 1., 1.], 2022-12-01T10:40:42.7076821Z [ 1., -1.], 2022-12-01T10:40:42.7077180Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7077576Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7077839Z tensor([[-1., 1.], 2022-12-01T10:40:42.7078031Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7078331Z [0., 0.]])} 2022-12-01T10:40:42.7078599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.7078958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.7079353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.7079707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.7080217Z 2022-12-01 10:14:04,566 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7080487Z [ 1., 1.], 2022-12-01T10:40:42.7080672Z [ 1., -1.], 2022-12-01T10:40:42.7081081Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7081485Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7081729Z tensor([[-1., 1.], 2022-12-01T10:40:42.7081933Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7082120Z [0., 0.]])} 2022-12-01T10:40:42.7082526Z 2022-12-01 10:14:04,566 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7082836Z [-1., -1.], 2022-12-01T10:40:42.7083019Z [-1., 1.], 2022-12-01T10:40:42.7083390Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7083801Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7084056Z tensor([[-1., 1.], 2022-12-01T10:40:42.7084256Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7084437Z [0., 0.]])} 2022-12-01T10:40:42.7084856Z 2022-12-01 10:14:04,566 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7085165Z [-1., 1.], 2022-12-01T10:40:42.7085339Z [-1., -1.], 2022-12-01T10:40:42.7085713Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7086119Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7086369Z tensor([[-1., 1.], 2022-12-01T10:40:42.7086562Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7086752Z [0., 0.]])} 2022-12-01T10:40:42.7087168Z 2022-12-01 10:14:04,566 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7087466Z [ 1., -1.], 2022-12-01T10:40:42.7087619Z [ 1., 1.], 2022-12-01T10:40:42.7088057Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7088459Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7088714Z tensor([[-1., 1.], 2022-12-01T10:40:42.7088921Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7089112Z [0., 0.]])} 2022-12-01T10:40:42.7089515Z 2022-12-01 10:14:04,587 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7089825Z [-1., 1.], 2022-12-01T10:40:42.7090010Z [-1., -1.], 2022-12-01T10:40:42.7090367Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7090767Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7091020Z tensor([[-1., 1.], 2022-12-01T10:40:42.7091125Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7091185Z [0., 0.]])} 2022-12-01T10:40:42.7091536Z 2022-12-01 10:14:04,587 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7091631Z [ 1., -1.], 2022-12-01T10:40:42.7091692Z [ 1., 1.], 2022-12-01T10:40:42.7091975Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7092182Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7092307Z tensor([[-1., 1.], 2022-12-01T10:40:42.7092411Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7092462Z [0., 0.]])} 2022-12-01T10:40:42.7092787Z 2022-12-01 10:14:04,587 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7092877Z [-1., -1.], 2022-12-01T10:40:42.7092966Z [-1., 1.], 2022-12-01T10:40:42.7093257Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7093467Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7093563Z tensor([[-1., 1.], 2022-12-01T10:40:42.7093669Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7093721Z [0., 0.]])} 2022-12-01T10:40:42.7094047Z 2022-12-01 10:14:04,587 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7094110Z [ 1., 1.], 2022-12-01T10:40:42.7094202Z [ 1., -1.], 2022-12-01T10:40:42.7094476Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7094687Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7094782Z tensor([[-1., 1.], 2022-12-01T10:40:42.7094887Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-12-01T10:40:42.7094939Z [0., 0.]])} 2022-12-01T10:40:42.7095200Z 2022-12-01 10:14:04,594 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-12-01T10:40:42.7095453Z 2022-12-01 10:14:04,594 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-12-01T10:40:42.7095706Z 2022-12-01 10:14:04,595 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-12-01T10:40:42.7095950Z 2022-12-01 10:14:04,596 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-12-01T10:40:42.7096194Z 2022-12-01 10:14:04,596 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-12-01T10:40:42.7096258Z ok (3.251s) 2022-12-01T10:40:42.7096265Z 2022-12-01T10:40:42.7096463Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7096526Z Ran 1 test in 3.251s 2022-12-01T10:40:42.7096540Z 2022-12-01T10:40:42.7096587Z OK 2022-12-01T10:40:42.7096592Z 2022-12-01T10:40:42.7096674Z Generating XML reports... 2022-12-01T10:40:42.7097074Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221201101401.xml 2022-12-01T10:40:42.7097348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7097468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7097749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7097880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7098098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4isz6pfq 2022-12-01T10:40:42.7098285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4isz6pfq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7098538Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7098543Z 2022-12-01T10:40:42.7098613Z Running tests... 2022-12-01T10:40:42.7098806Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7099106Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12246 2022-12-01T10:40:42.7099263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12247 2022-12-01T10:40:42.7099415Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12248 2022-12-01T10:40:42.7099568Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12249 2022-12-01T10:40:42.7099718Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 12250 2022-12-01T10:40:42.7099856Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 12251 2022-12-01T10:40:42.7100128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7100247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7100533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7100670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7100858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpep4ya5me 2022-12-01T10:40:42.7101052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpep4ya5me/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7101323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7101434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7101710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7101843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7102028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejtovram 2022-12-01T10:40:42.7102226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejtovram/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7102500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7102623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7102905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7103041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7103213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxjbh6rv 2022-12-01T10:40:42.7103406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxjbh6rv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7103568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7103734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7104004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7104126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7104404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7104563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7104737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwk3q53x 2022-12-01T10:40:42.7104928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwk3q53x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7105195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7105342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7105619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7105753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7105936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjo4h1hm 2022-12-01T10:40:42.7106206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7106331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7106597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7106727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7106909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8ah3pcq 2022-12-01T10:40:42.7107102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjo4h1hm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7107293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8ah3pcq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7107456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7107617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7107781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-12-01T10:40:42.7107943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-12-01T10:40:42.7108197Z 2022-12-01 10:14:08,836 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-12-01T10:40:42.7108519Z 2022-12-01 10:14:08,837 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7108780Z 2022-12-01 10:14:08,856 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-12-01T10:40:42.7109098Z 2022-12-01 10:14:08,856 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7109358Z 2022-12-01 10:14:08,866 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-12-01T10:40:42.7109673Z 2022-12-01 10:14:08,866 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7109929Z 2022-12-01 10:14:08,876 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-12-01T10:40:42.7110192Z 2022-12-01 10:14:08,876 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-12-01T10:40:42.7110503Z 2022-12-01 10:14:08,876 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7110765Z 2022-12-01 10:14:08,877 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-12-01T10:40:42.7110931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7111134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7111308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7111481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7111650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-12-01T10:40:42.7111856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-12-01T10:40:42.7112155Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7112454Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7112749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7113033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7113320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7113606Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7113900Z 2022-12-01 10:14:08,930 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-12-01T10:40:42.7114074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-12-01T10:40:42.7114356Z 2022-12-01 10:14:08,930 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-12-01T10:40:42.7114617Z 2022-12-01 10:14:08,930 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-12-01T10:40:42.7114943Z 2022-12-01 10:14:08,931 ddp_under_dist_autograd_test.py:396 INFO p:process 5 t:MainThread: Running DDP + RPC test with simulating uneven inputs across trainers. 2022-12-01T10:40:42.7115115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-12-01T10:40:42.7115400Z 2022-12-01 10:14:08,940 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-12-01T10:40:42.7115674Z 2022-12-01 10:14:08,940 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-12-01T10:40:42.7115924Z 2022-12-01 10:14:08,952 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-12-01T10:40:42.7116179Z 2022-12-01 10:14:08,952 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-12-01T10:40:42.7116355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-12-01T10:40:42.7116530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-12-01T10:40:42.7116702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-12-01T10:40:42.7116871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-12-01T10:40:42.7117170Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7117463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7117742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7118061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7118541Z 2022-12-01 10:14:09,102 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7118821Z 2022-12-01 10:14:09,102 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-12-01T10:40:42.7119114Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7119457Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-12-01T10:40:42.7119732Z 2022-12-01 10:14:09,112 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7120012Z 2022-12-01 10:14:09,112 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7120291Z 2022-12-01 10:14:09,112 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-12-01T10:40:42.7120566Z 2022-12-01 10:14:09,112 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-12-01T10:40:42.7120826Z 2022-12-01 10:14:09,120 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7121106Z 2022-12-01 10:14:09,120 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-12-01T10:40:42.7121521Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.7121664Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.7122016Z 2022-12-01 10:14:09,151 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-12-01T10:40:42.7122425Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.7122566Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.7122924Z 2022-12-01 10:14:09,152 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-12-01T10:40:42.7123190Z 2022-12-01 10:14:09,153 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-3: Trainer reduced input patches from 2 2022-12-01T10:40:42.7123291Z to 1 to simulate uneven inputs. 2022-12-01T10:40:42.7123698Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.7123826Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.7124171Z 2022-12-01 10:14:09,153 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-12-01T10:40:42.7124577Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-12-01T10:40:42.7124718Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-12-01T10:40:42.7125066Z 2022-12-01 10:14:09,154 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-12-01T10:40:42.7125379Z 2022-12-01 10:14:09,155 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-3: Trainer reduced input patches from 2 2022-12-01T10:40:42.7125477Z to 1 to simulate uneven inputs. 2022-12-01T10:40:42.7125797Z 2022-12-01 10:14:09,202 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7125887Z [ 1., -1.], 2022-12-01T10:40:42.7125938Z [ 1., 1.], 2022-12-01T10:40:42.7126231Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7126476Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7126574Z tensor([[-1., 1.], 2022-12-01T10:40:42.7126684Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7126751Z [ 0., 4.]])} 2022-12-01T10:40:42.7127073Z 2022-12-01 10:14:09,205 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7127136Z [ 1., 1.], 2022-12-01T10:40:42.7127216Z [ 1., -1.], 2022-12-01T10:40:42.7127494Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7127760Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7127865Z tensor([[-1., 1.], 2022-12-01T10:40:42.7127973Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7128040Z [ 0., 4.]])} 2022-12-01T10:40:42.7128367Z 2022-12-01 10:14:09,205 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7128461Z [-1., -1.], 2022-12-01T10:40:42.7128539Z [-1., 1.], 2022-12-01T10:40:42.7128831Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7129044Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7129139Z tensor([[-1., 1.], 2022-12-01T10:40:42.7129246Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7129313Z [ 0., 4.]])} 2022-12-01T10:40:42.7129629Z 2022-12-01 10:14:09,205 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7129727Z [-1., 1.], 2022-12-01T10:40:42.7129807Z [-1., -1.], 2022-12-01T10:40:42.7130085Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7130300Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7130398Z tensor([[-1., 1.], 2022-12-01T10:40:42.7130506Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7130570Z [ 0., 4.]])} 2022-12-01T10:40:42.7130742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.7130898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:40:42.7131213Z 2022-12-01 10:14:09,222 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7131307Z [-1., 1.], 2022-12-01T10:40:42.7131395Z [-1., -1.], 2022-12-01T10:40:42.7131675Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7131884Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7132015Z tensor([[-1., 1.], 2022-12-01T10:40:42.7132174Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-12-01T10:40:42.7132256Z [-2., 2.]])} 2022-12-01T10:40:42.7132579Z 2022-12-01 10:14:09,222 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7132646Z [ 1., 1.], 2022-12-01T10:40:42.7132736Z [ 1., -1.], 2022-12-01T10:40:42.7133042Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7133253Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7133346Z tensor([[-1., 1.], 2022-12-01T10:40:42.7133507Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-12-01T10:40:42.7133590Z [-2., 2.]])} 2022-12-01T10:40:42.7133864Z 2022-12-01 10:14:09,225 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-4: Trainer reduced input patches from 2 2022-12-01T10:40:42.7133967Z to 1 to simulate uneven inputs. 2022-12-01T10:40:42.7134235Z 2022-12-01 10:14:09,225 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-4: Trainer reduced input patches from 2 2022-12-01T10:40:42.7134334Z to 1 to simulate uneven inputs. 2022-12-01T10:40:42.7134658Z 2022-12-01 10:14:09,238 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7134755Z [ 1., -1.], 2022-12-01T10:40:42.7134820Z [ 1., 1.], 2022-12-01T10:40:42.7135098Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7135318Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7135417Z tensor([[-1., 1.], 2022-12-01T10:40:42.7135524Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7135590Z [ 0., 4.]])} 2022-12-01T10:40:42.7135907Z 2022-12-01 10:14:09,238 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7136000Z [-1., 1.], 2022-12-01T10:40:42.7136087Z [-1., -1.], 2022-12-01T10:40:42.7136373Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7136587Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7136683Z tensor([[-1., 1.], 2022-12-01T10:40:42.7136792Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7136846Z [ 0., 4.]])} 2022-12-01T10:40:42.7137166Z 2022-12-01 10:14:09,238 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7137260Z [-1., -1.], 2022-12-01T10:40:42.7137348Z [-1., 1.], 2022-12-01T10:40:42.7137633Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7137847Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7137946Z tensor([[-1., 1.], 2022-12-01T10:40:42.7138054Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7138108Z [ 0., 4.]])} 2022-12-01T10:40:42.7138428Z 2022-12-01 10:14:09,238 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7138490Z [ 1., 1.], 2022-12-01T10:40:42.7138580Z [ 1., -1.], 2022-12-01T10:40:42.7138881Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7139091Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7139187Z tensor([[-1., 1.], 2022-12-01T10:40:42.7139293Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7139348Z [ 0., 4.]])} 2022-12-01T10:40:42.7139700Z 2022-12-01 10:14:09,250 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7139761Z [ 1., 1.], 2022-12-01T10:40:42.7139851Z [ 1., -1.], 2022-12-01T10:40:42.7140126Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7140335Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7140432Z tensor([[-1., 1.], 2022-12-01T10:40:42.7140593Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-12-01T10:40:42.7140675Z [-2., 2.]])} 2022-12-01T10:40:42.7140991Z 2022-12-01 10:14:09,250 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7141082Z [-1., 1.], 2022-12-01T10:40:42.7141174Z [-1., -1.], 2022-12-01T10:40:42.7141454Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7141663Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7141759Z tensor([[-1., 1.], 2022-12-01T10:40:42.7141922Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-12-01T10:40:42.7142006Z [-2., 2.]])} 2022-12-01T10:40:42.7142281Z 2022-12-01 10:14:09,253 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-5: Trainer reduced input patches from 2 2022-12-01T10:40:42.7142381Z to 1 to simulate uneven inputs. 2022-12-01T10:40:42.7142652Z 2022-12-01 10:14:09,253 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-5: Trainer reduced input patches from 2 2022-12-01T10:40:42.7142748Z to 1 to simulate uneven inputs. 2022-12-01T10:40:42.7143072Z 2022-12-01 10:14:09,271 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7143169Z [ 1., -1.], 2022-12-01T10:40:42.7143221Z [ 1., 1.], 2022-12-01T10:40:42.7143509Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7143723Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7143822Z tensor([[-1., 1.], 2022-12-01T10:40:42.7143928Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7143996Z [ 0., 4.]])} 2022-12-01T10:40:42.7144313Z 2022-12-01 10:14:09,271 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7144378Z [ 1., 1.], 2022-12-01T10:40:42.7144457Z [ 1., -1.], 2022-12-01T10:40:42.7144739Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7144955Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7145053Z tensor([[-1., 1.], 2022-12-01T10:40:42.7145160Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7145227Z [ 0., 4.]])} 2022-12-01T10:40:42.7145576Z 2022-12-01 10:14:09,271 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7145671Z [-1., 1.], 2022-12-01T10:40:42.7145752Z [-1., -1.], 2022-12-01T10:40:42.7146029Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7146242Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7146366Z tensor([[-1., 1.], 2022-12-01T10:40:42.7146473Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7146540Z [ 0., 4.]])} 2022-12-01T10:40:42.7146862Z 2022-12-01 10:14:09,273 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7146958Z [-1., -1.], 2022-12-01T10:40:42.7147039Z [-1., 1.], 2022-12-01T10:40:42.7147327Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7147543Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-12-01T10:40:42.7147641Z tensor([[-1., 1.], 2022-12-01T10:40:42.7147750Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-12-01T10:40:42.7147818Z [ 0., 4.]])} 2022-12-01T10:40:42.7148145Z 2022-12-01 10:14:09,284 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7148211Z [ 1., 1.], 2022-12-01T10:40:42.7148290Z [ 1., -1.], 2022-12-01T10:40:42.7148565Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7148779Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7148877Z tensor([[-1., 1.], 2022-12-01T10:40:42.7149039Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-12-01T10:40:42.7149133Z [-2., 2.]])} 2022-12-01T10:40:42.7149457Z 2022-12-01 10:14:09,284 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7149550Z [-1., 1.], 2022-12-01T10:40:42.7149631Z [-1., -1.], 2022-12-01T10:40:42.7149910Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7150119Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-12-01T10:40:42.7150217Z tensor([[-1., 1.], 2022-12-01T10:40:42.7150380Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-12-01T10:40:42.7150474Z [-2., 2.]])} 2022-12-01T10:40:42.7150742Z 2022-12-01 10:14:09,292 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-12-01T10:40:42.7150985Z 2022-12-01 10:14:09,293 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-12-01T10:40:42.7151241Z 2022-12-01 10:14:09,293 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-12-01T10:40:42.7151491Z 2022-12-01 10:14:09,294 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-12-01T10:40:42.7151742Z 2022-12-01 10:14:09,294 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-12-01T10:40:42.7152042Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-12-01T10:40:42.7152159Z _warnings.warn(warn_message, ResourceWarning) 2022-12-01T10:40:42.7152485Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-12-01T10:40:42.7152599Z _warnings.warn(warn_message, ResourceWarning) 2022-12-01T10:40:42.7152886Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-12-01T10:40:42.7152985Z _warnings.warn(warn_message, ResourceWarning) 2022-12-01T10:40:42.7153270Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-12-01T10:40:42.7153408Z _warnings.warn(warn_message, ResourceWarning) 2022-12-01T10:40:42.7153478Z ok (3.247s) 2022-12-01T10:40:42.7153485Z 2022-12-01T10:40:42.7153686Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7153761Z Ran 1 test in 3.248s 2022-12-01T10:40:42.7153766Z 2022-12-01T10:40:42.7153826Z OK 2022-12-01T10:40:42.7153831Z 2022-12-01T10:40:42.7153916Z Generating XML reports... 2022-12-01T10:40:42.7154306Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221201101406.xml 2022-12-01T10:40:42.7154583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7154707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7154991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7155130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7155315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdj3kelk 2022-12-01T10:40:42.7155512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdj3kelk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7155769Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7155774Z 2022-12-01T10:40:42.7155847Z Running tests... 2022-12-01T10:40:42.7156032Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7156287Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12660 2022-12-01T10:40:42.7156448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12661 2022-12-01T10:40:42.7156603Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12662 2022-12-01T10:40:42.7156758Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12663 2022-12-01T10:40:42.7156909Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 12664 2022-12-01T10:40:42.7157060Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 12665 2022-12-01T10:40:42.7157336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7157449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7157732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7157867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7158051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t5fjy4q 2022-12-01T10:40:42.7158432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t5fjy4q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7158715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7158840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7159120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7159308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7159484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cwkr1ko 2022-12-01T10:40:42.7159682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cwkr1ko/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7159954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7160078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7160399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7160535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7160721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidl26e3l 2022-12-01T10:40:42.7160920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidl26e3l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7161194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7161305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7161584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7161717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7161904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnpyvdmz 2022-12-01T10:40:42.7162100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnpyvdmz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7162263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-12-01T10:40:42.7162428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7162703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7162813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7163094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7163230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7163414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fxg0e0o 2022-12-01T10:40:42.7163612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fxg0e0o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7163881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7164001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7164282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7164416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7164587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_o_n03v 2022-12-01T10:40:42.7164777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_o_n03v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7164945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7165110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7165271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-12-01T10:40:42.7165432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7165699Z 2022-12-01 10:14:13,589 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-12-01T10:40:42.7166057Z 2022-12-01 10:14:13,589 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7166323Z 2022-12-01 10:14:13,599 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-12-01T10:40:42.7166630Z 2022-12-01 10:14:13,599 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7166925Z 2022-12-01 10:14:13,609 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-12-01T10:40:42.7167245Z 2022-12-01 10:14:13,609 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7167502Z 2022-12-01 10:14:13,619 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-12-01T10:40:42.7167823Z 2022-12-01 10:14:13,619 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-12-01T10:40:42.7168151Z 2022-12-01 10:14:13,619 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-12-01T10:40:42.7168420Z 2022-12-01 10:14:13,619 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-12-01T10:40:42.7168603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7168783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7168961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7169123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7169299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-12-01T10:40:42.7169472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-12-01T10:40:42.7169778Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7170079Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7170380Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7170677Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7170969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7171266Z 2022-12-01 10:14:13,682 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-12-01T10:40:42.7171544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-12-01T10:40:42.7171834Z 2022-12-01 10:14:13,682 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-12-01T10:40:42.7172127Z 2022-12-01 10:14:13,683 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-12-01T10:40:42.7172415Z 2022-12-01 10:14:13,683 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-12-01T10:40:42.7172678Z 2022-12-01 10:14:13,683 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-12-01T10:40:42.7173003Z 2022-12-01 10:14:13,688 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7173354Z 2022-12-01 10:14:13,688 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-12-01T10:40:42.7173612Z 2022-12-01 10:14:13,714 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-12-01T10:40:42.7173871Z 2022-12-01 10:14:13,714 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-12-01T10:40:42.7174220Z 2022-12-01 10:14:13,739 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7174305Z [ 1., -1.], 2022-12-01T10:40:42.7174370Z [ 1., 1.], 2022-12-01T10:40:42.7174663Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7174891Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-12-01T10:40:42.7174988Z tensor([[-1., 1.], 2022-12-01T10:40:42.7175156Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-12-01T10:40:42.7175258Z [ 0., -16.]])} 2022-12-01T10:40:42.7175538Z 2022-12-01 10:14:13,762 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7175881Z 2022-12-01 10:14:13,762 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-12-01T10:40:42.7176161Z 2022-12-01 10:14:13,818 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7176514Z 2022-12-01 10:14:13,818 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-12-01T10:40:42.7176835Z 2022-12-01 10:14:13,831 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7176929Z [-1., -1.], 2022-12-01T10:40:42.7177020Z [-1., 1.], 2022-12-01T10:40:42.7177314Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7177537Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-12-01T10:40:42.7177634Z tensor([[-1., 1.], 2022-12-01T10:40:42.7177785Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-12-01T10:40:42.7177884Z [ 16., -16.]])} 2022-12-01T10:40:42.7178206Z 2022-12-01 10:14:13,837 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7178300Z [-1., 1.], 2022-12-01T10:40:42.7178392Z [-1., -1.], 2022-12-01T10:40:42.7178672Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7178890Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-12-01T10:40:42.7178988Z tensor([[-1., 1.], 2022-12-01T10:40:42.7179141Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-12-01T10:40:42.7179239Z [-16., 16.]])} 2022-12-01T10:40:42.7179516Z 2022-12-01 10:14:13,845 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-12-01T10:40:42.7179871Z 2022-12-01 10:14:13,845 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-12-01T10:40:42.7180228Z 2022-12-01 10:14:13,866 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7180294Z [ 1., 1.], 2022-12-01T10:40:42.7180387Z [ 1., -1.], 2022-12-01T10:40:42.7180667Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7180870Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-12-01T10:40:42.7180995Z tensor([[-1., 1.], 2022-12-01T10:40:42.7181105Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-12-01T10:40:42.7181173Z [ 0., 16.]])} 2022-12-01T10:40:42.7181493Z 2022-12-01 10:14:13,879 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7181587Z [-1., 1.], 2022-12-01T10:40:42.7181677Z [-1., -1.], 2022-12-01T10:40:42.7181957Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7182160Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-12-01T10:40:42.7182257Z tensor([[-1., 1.], 2022-12-01T10:40:42.7182420Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-12-01T10:40:42.7182517Z [-16., 16.]])} 2022-12-01T10:40:42.7182838Z 2022-12-01 10:14:13,885 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7182937Z [ 1., -1.], 2022-12-01T10:40:42.7183002Z [ 1., 1.], 2022-12-01T10:40:42.7183290Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7183499Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-12-01T10:40:42.7183603Z tensor([[-1., 1.], 2022-12-01T10:40:42.7183768Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-12-01T10:40:42.7183868Z [ 0., -16.]])} 2022-12-01T10:40:42.7184195Z 2022-12-01 10:14:13,885 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7184289Z [-1., -1.], 2022-12-01T10:40:42.7184383Z [-1., 1.], 2022-12-01T10:40:42.7184672Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7184882Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-12-01T10:40:42.7184979Z tensor([[-1., 1.], 2022-12-01T10:40:42.7185144Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-12-01T10:40:42.7185289Z [ 16., -16.]])} 2022-12-01T10:40:42.7185629Z 2022-12-01 10:14:13,891 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7185811Z [ 1., 1.], 2022-12-01T10:40:42.7185940Z [ 1., -1.], 2022-12-01T10:40:42.7186253Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7186459Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-12-01T10:40:42.7186591Z tensor([[-1., 1.], 2022-12-01T10:40:42.7186751Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-12-01T10:40:42.7186849Z [ 0., 16.]])} 2022-12-01T10:40:42.7187199Z 2022-12-01 10:14:13,906 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-12-01T10:40:42.7187297Z [ 1., 1.], 2022-12-01T10:40:42.7187485Z [ 1., -1.], 2022-12-01T10:40:42.7187803Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7188010Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-12-01T10:40:42.7188152Z tensor([[-1., 1.], 2022-12-01T10:40:42.7188288Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-12-01T10:40:42.7188420Z [ 0., 16.]])} 2022-12-01T10:40:42.7188780Z 2022-12-01 10:14:13,908 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-12-01T10:40:42.7188937Z [-1., -1.], 2022-12-01T10:40:42.7189057Z [-1., 1.], 2022-12-01T10:40:42.7189403Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7189617Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-12-01T10:40:42.7189752Z tensor([[-1., 1.], 2022-12-01T10:40:42.7189946Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-12-01T10:40:42.7190072Z [ 16., -16.]])} 2022-12-01T10:40:42.7190429Z 2022-12-01 10:14:13,909 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-12-01T10:40:42.7190555Z [-1., 1.], 2022-12-01T10:40:42.7190676Z [-1., -1.], 2022-12-01T10:40:42.7190943Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7191185Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-12-01T10:40:42.7191338Z tensor([[-1., 1.], 2022-12-01T10:40:42.7191535Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-12-01T10:40:42.7191665Z [-16., 16.]])} 2022-12-01T10:40:42.7192014Z 2022-12-01 10:14:13,909 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-12-01T10:40:42.7192136Z [ 1., -1.], 2022-12-01T10:40:42.7192257Z [ 1., 1.], 2022-12-01T10:40:42.7192534Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-12-01T10:40:42.7192786Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-12-01T10:40:42.7192921Z tensor([[-1., 1.], 2022-12-01T10:40:42.7193136Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-12-01T10:40:42.7193267Z [ 0., -16.]])} 2022-12-01T10:40:42.7193567Z 2022-12-01 10:14:13,916 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-12-01T10:40:42.7193854Z 2022-12-01 10:14:13,917 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-12-01T10:40:42.7194133Z 2022-12-01 10:14:13,918 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-12-01T10:40:42.7194378Z 2022-12-01 10:14:13,918 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-12-01T10:40:42.7194658Z 2022-12-01 10:14:13,919 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-12-01T10:40:42.7194763Z ok (3.246s) 2022-12-01T10:40:42.7194770Z 2022-12-01T10:40:42.7194997Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7195122Z Ran 1 test in 3.246s 2022-12-01T10:40:42.7195127Z 2022-12-01T10:40:42.7195215Z OK 2022-12-01T10:40:42.7195220Z 2022-12-01T10:40:42.7195334Z Generating XML reports... 2022-12-01T10:40:42.7195806Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221201101411.xml 2022-12-01T10:40:42.7196160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7196272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7196585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7196750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7196996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zk79o65 2022-12-01T10:40:42.7197260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zk79o65/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7197546Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7197551Z 2022-12-01T10:40:42.7197653Z Running tests... 2022-12-01T10:40:42.7197889Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7198020Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.7198415Z This test ensures async processing for distributed autograd works ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13062 2022-12-01T10:40:42.7198603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13063 2022-12-01T10:40:42.7198786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13064 2022-12-01T10:40:42.7198972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13065 2022-12-01T10:40:42.7199302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7199456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7199780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7199973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7200149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk614age1 2022-12-01T10:40:42.7200372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk614age1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7200679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7200833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7201147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7201333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7201558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wflfser 2022-12-01T10:40:42.7201788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wflfser/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7202090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7202201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7202512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7202679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7202892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp052h_vn3 2022-12-01T10:40:42.7203117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp052h_vn3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7203447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7203650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7203993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7204117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7204331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9_l95mc 2022-12-01T10:40:42.7204552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9_l95mc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7204781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7204982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7205174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7205385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7205594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7205799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7205961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7206162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7206496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7206832Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7207155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7207479Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7207596Z ok (2.238s) 2022-12-01T10:40:42.7207603Z 2022-12-01T10:40:42.7207941Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7208046Z Ran 1 test in 2.238s 2022-12-01T10:40:42.7208052Z 2022-12-01T10:40:42.7208099Z OK 2022-12-01T10:40:42.7208104Z 2022-12-01T10:40:42.7208217Z Generating XML reports... 2022-12-01T10:40:42.7208632Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101415.xml 2022-12-01T10:40:42.7208934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7209088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7209396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7209586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7209803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvs85oxz 2022-12-01T10:40:42.7209990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvs85oxz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7210276Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7210283Z 2022-12-01T10:40:42.7210394Z Running tests... 2022-12-01T10:40:42.7210621Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7210890Z test_autograd_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13377 2022-12-01T10:40:42.7211075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13378 2022-12-01T10:40:42.7211290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13379 2022-12-01T10:40:42.7211530Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13380 2022-12-01T10:40:42.7211839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7211949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7212261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7212454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7212670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbe1tl1y 2022-12-01T10:40:42.7212894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbe1tl1y/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7213196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7213353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7213688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7213854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7214026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79y2skn8 2022-12-01T10:40:42.7214252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79y2skn8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7214555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7214708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7215016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7215181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7215430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeikau_j_ 2022-12-01T10:40:42.7215675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeikau_j_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7215938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7216093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7216398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7216561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7216773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4_6ejqo 2022-12-01T10:40:42.7216997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4_6ejqo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7217204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7217399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7217605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7217773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7217872Z ok (2.539s) 2022-12-01T10:40:42.7217877Z 2022-12-01T10:40:42.7218109Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7218212Z Ran 1 test in 2.539s 2022-12-01T10:40:42.7218217Z 2022-12-01T10:40:42.7218305Z OK 2022-12-01T10:40:42.7218310Z 2022-12-01T10:40:42.7218430Z Generating XML reports... 2022-12-01T10:40:42.7218826Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101419.xml 2022-12-01T10:40:42.7219184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7219296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7219626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7219789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7220034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4jkk4ef 2022-12-01T10:40:42.7220267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4jkk4ef/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7220550Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7220556Z 2022-12-01T10:40:42.7220657Z Running tests... 2022-12-01T10:40:42.7220882Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7221165Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13600 2022-12-01T10:40:42.7221313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13601 2022-12-01T10:40:42.7221516Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13602 2022-12-01T10:40:42.7221697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13603 2022-12-01T10:40:42.7222010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7222161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7222468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7222658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7222876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec750zf_ 2022-12-01T10:40:42.7223059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec750zf_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7223362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7223530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7223849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7224013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7224225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5856u4ql 2022-12-01T10:40:42.7224447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5856u4ql/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7224747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7224901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7225166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7225331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7225577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyypqm1_ 2022-12-01T10:40:42.7225800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyypqm1_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7226101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7226256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7226612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7226776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7226998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp397w9cpt 2022-12-01T10:40:42.7227177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp397w9cpt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7227367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7227630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7227822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7228012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7228106Z ok (2.039s) 2022-12-01T10:40:42.7228112Z 2022-12-01T10:40:42.7228341Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7228452Z Ran 1 test in 2.039s 2022-12-01T10:40:42.7228457Z 2022-12-01T10:40:42.7228505Z OK 2022-12-01T10:40:42.7228510Z 2022-12-01T10:40:42.7228621Z Generating XML reports... 2022-12-01T10:40:42.7229019Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101423.xml 2022-12-01T10:40:42.7229336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7229491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7229797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7229980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7230198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbh63cgt2 2022-12-01T10:40:42.7230422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbh63cgt2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7230664Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7230670Z 2022-12-01T10:40:42.7230771Z Running tests... 2022-12-01T10:40:42.7230994Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7231293Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13843 2022-12-01T10:40:42.7231482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13844 2022-12-01T10:40:42.7231669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13845 2022-12-01T10:40:42.7231859Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13846 2022-12-01T10:40:42.7232161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7232272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7232583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7232745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7232963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0x2barxw 2022-12-01T10:40:42.7233209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0x2barxw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7233512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7233671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7234029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7234192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7234363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xau2ac5 2022-12-01T10:40:42.7234587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xau2ac5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7234885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7235066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7235402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7235565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7235779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskk58mi_ 2022-12-01T10:40:42.7236002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskk58mi_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7236262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7236413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7236718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7236884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7237104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00zhrehq 2022-12-01T10:40:42.7237346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00zhrehq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7237565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7237755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7237945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7238095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7238296Z ok (2.240s) 2022-12-01T10:40:42.7238303Z 2022-12-01T10:40:42.7238536Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7238649Z Ran 1 test in 2.240s 2022-12-01T10:40:42.7238655Z 2022-12-01T10:40:42.7238742Z OK 2022-12-01T10:40:42.7238747Z 2022-12-01T10:40:42.7238879Z Generating XML reports... 2022-12-01T10:40:42.7239282Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101426.xml 2022-12-01T10:40:42.7239588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7303138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7303657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7303795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7303977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13ktq944 2022-12-01T10:40:42.7304187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13ktq944/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7304455Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7304463Z 2022-12-01T10:40:42.7304534Z Running tests... 2022-12-01T10:40:42.7304735Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7304989Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14086 2022-12-01T10:40:42.7305272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14087 2022-12-01T10:40:42.7305434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14088 2022-12-01T10:40:42.7305587Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14089 2022-12-01T10:40:42.7305857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7306028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7306308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7306441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7306627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wje_hef 2022-12-01T10:40:42.7306818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wje_hef/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7307092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7307215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7307494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7307619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7307805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprh0mweg2 2022-12-01T10:40:42.7308002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprh0mweg2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7308275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7308394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7308675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7308810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7308992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3atancn9 2022-12-01T10:40:42.7309175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3atancn9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7309452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7309569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7309879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7310012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7310213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fvw6o6v 2022-12-01T10:40:42.7310407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fvw6o6v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7310570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7310751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7310921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7311087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7311149Z ok (2.138s) 2022-12-01T10:40:42.7311155Z 2022-12-01T10:40:42.7311352Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7311441Z Ran 1 test in 2.138s 2022-12-01T10:40:42.7311447Z 2022-12-01T10:40:42.7311507Z OK 2022-12-01T10:40:42.7311511Z 2022-12-01T10:40:42.7311647Z Generating XML reports... 2022-12-01T10:40:42.7312041Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101430.xml 2022-12-01T10:40:42.7312321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7312442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7312778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7312913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7313111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_4twsgp 2022-12-01T10:40:42.7313308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_4twsgp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7313563Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7313569Z 2022-12-01T10:40:42.7313640Z Running tests... 2022-12-01T10:40:42.7313836Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7314074Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14329 2022-12-01T10:40:42.7314228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14330 2022-12-01T10:40:42.7314384Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14331 2022-12-01T10:40:42.7314536Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14332 2022-12-01T10:40:42.7314812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7314929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7315210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7315344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7315527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdoucyf_j 2022-12-01T10:40:42.7315712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdoucyf_j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7315986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7316109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7316388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7316521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7316710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvy7zxwq 2022-12-01T10:40:42.7316900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvy7zxwq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7317166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7317277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7317555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7317692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7317871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvq_1kmuk 2022-12-01T10:40:42.7318061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvq_1kmuk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7318549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7318674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7318956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7319085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7319259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6mrnbzb 2022-12-01T10:40:42.7319489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6mrnbzb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7319654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7319818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7319979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7320139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7320206Z ok (2.139s) 2022-12-01T10:40:42.7320212Z 2022-12-01T10:40:42.7320409Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7320474Z Ran 1 test in 2.139s 2022-12-01T10:40:42.7320479Z 2022-12-01T10:40:42.7320534Z OK 2022-12-01T10:40:42.7320539Z 2022-12-01T10:40:42.7320622Z Generating XML reports... 2022-12-01T10:40:42.7320994Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101434.xml 2022-12-01T10:40:42.7321262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7321386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7321663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7321798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7321979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnfef79h 2022-12-01T10:40:42.7322164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnfef79h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7322419Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7322426Z 2022-12-01T10:40:42.7322496Z Running tests... 2022-12-01T10:40:42.7322692Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7322941Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14572 2022-12-01T10:40:42.7323098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14573 2022-12-01T10:40:42.7323257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14574 2022-12-01T10:40:42.7323404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14575 2022-12-01T10:40:42.7323670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7323793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7324075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7324213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7324399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcrsania 2022-12-01T10:40:42.7324593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcrsania/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7324868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7325019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7325302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7325427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7325609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvirbfmx 2022-12-01T10:40:42.7325834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvirbfmx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7326101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7326220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7326499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7326633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7326817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa63q06ll 2022-12-01T10:40:42.7326999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa63q06ll/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7327287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7327412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7327748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7327888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7328099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ejgg7r2 2022-12-01T10:40:42.7328304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ejgg7r2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7328548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7333904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7334080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7334245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7334317Z ok (2.237s) 2022-12-01T10:40:42.7334323Z 2022-12-01T10:40:42.7334535Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7334607Z Ran 1 test in 2.237s 2022-12-01T10:40:42.7334612Z 2022-12-01T10:40:42.7334672Z OK 2022-12-01T10:40:42.7334677Z 2022-12-01T10:40:42.7334759Z Generating XML reports... 2022-12-01T10:40:42.7335133Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101437.xml 2022-12-01T10:40:42.7335412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7335523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7335803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7335938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7336127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahwh85wp 2022-12-01T10:40:42.7336326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahwh85wp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7336577Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7336583Z 2022-12-01T10:40:42.7336655Z Running tests... 2022-12-01T10:40:42.7336893Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7337125Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14815 2022-12-01T10:40:42.7337281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14816 2022-12-01T10:40:42.7337429Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14817 2022-12-01T10:40:42.7358171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14818 2022-12-01T10:40:42.7358663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7358780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7359061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7359189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7359370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy300qtg 2022-12-01T10:40:42.7359555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy300qtg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7359822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7359936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7360213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7360339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7360514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx4ob_uf 2022-12-01T10:40:42.7360779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7360895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7361166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7361291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7361466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpzgvhe2 2022-12-01T10:40:42.7361647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx4ob_uf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7361835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpzgvhe2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7362098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7362214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7362488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7362614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7362791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujpsezvc 2022-12-01T10:40:42.7362977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujpsezvc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7363132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7363285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7363440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7363593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7363656Z ok (2.140s) 2022-12-01T10:40:42.7363661Z 2022-12-01T10:40:42.7363915Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7363985Z Ran 1 test in 2.141s 2022-12-01T10:40:42.7363991Z 2022-12-01T10:40:42.7364046Z OK 2022-12-01T10:40:42.7364051Z 2022-12-01T10:40:42.7364130Z Generating XML reports... 2022-12-01T10:40:42.7364490Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101441.xml 2022-12-01T10:40:42.7364758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7364903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7365172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7365300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7365478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0laruobo 2022-12-01T10:40:42.7365668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0laruobo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7365917Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7365922Z 2022-12-01T10:40:42.7365989Z Running tests... 2022-12-01T10:40:42.7366173Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7366886Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77516 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.217s) 2022-12-01T10:40:42.7366901Z 2022-12-01T10:40:42.7367082Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7367151Z Ran 1 test in 0.217s 2022-12-01T10:40:42.7367158Z 2022-12-01T10:40:42.7367224Z OK (skipped=1) 2022-12-01T10:40:42.7367229Z 2022-12-01T10:40:42.7367306Z Generating XML reports... 2022-12-01T10:40:42.7367665Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101445.xml 2022-12-01T10:40:42.7368017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7368138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7368411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7368532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7368713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatecmnvs 2022-12-01T10:40:42.7368904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatecmnvs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7369150Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7369155Z 2022-12-01T10:40:42.7369222Z Running tests... 2022-12-01T10:40:42.7369410Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7369648Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15052 2022-12-01T10:40:42.7369801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15053 2022-12-01T10:40:42.7369950Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15054 2022-12-01T10:40:42.7370092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15055 2022-12-01T10:40:42.7370360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7370512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7370791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7370920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7371100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2q3trnpu 2022-12-01T10:40:42.7371291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2q3trnpu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7371583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7371694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7371968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7372098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7372278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50w8falk 2022-12-01T10:40:42.7372468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50w8falk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7372732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7372849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7373125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7373253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7373424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kuct5ca 2022-12-01T10:40:42.7373614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kuct5ca/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7373880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7373998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7374269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7374396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7374574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hs2v6zk 2022-12-01T10:40:42.7374761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hs2v6zk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7374918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7375069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7375227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7375384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7375446Z ok (2.138s) 2022-12-01T10:40:42.7375451Z 2022-12-01T10:40:42.7375641Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7375711Z Ran 1 test in 2.139s 2022-12-01T10:40:42.7375716Z 2022-12-01T10:40:42.7375769Z OK 2022-12-01T10:40:42.7375776Z 2022-12-01T10:40:42.7375853Z Generating XML reports... 2022-12-01T10:40:42.7376210Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101446.xml 2022-12-01T10:40:42.7376474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7376593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7376898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7377028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7377208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1140lkde 2022-12-01T10:40:42.7377395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1140lkde/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7377672Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7377678Z 2022-12-01T10:40:42.7377744Z Running tests... 2022-12-01T10:40:42.7377928Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7378630Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77453 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.217s) 2022-12-01T10:40:42.7378636Z 2022-12-01T10:40:42.7378824Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7378894Z Ran 1 test in 0.218s 2022-12-01T10:40:42.7378899Z 2022-12-01T10:40:42.7378965Z OK (skipped=1) 2022-12-01T10:40:42.7378970Z 2022-12-01T10:40:42.7379048Z Generating XML reports... 2022-12-01T10:40:42.7379413Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101450.xml 2022-12-01T10:40:42.7379679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7379795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7380074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7380196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7380377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3kzumoo 2022-12-01T10:40:42.7380568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3kzumoo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7380817Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7380825Z 2022-12-01T10:40:42.7380890Z Running tests... 2022-12-01T10:40:42.7381078Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7381317Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15305 2022-12-01T10:40:42.7381468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15306 2022-12-01T10:40:42.7381614Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15307 2022-12-01T10:40:42.7381761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15308 2022-12-01T10:40:42.7382029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7382148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7382421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7382552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7382730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03c16f_3 2022-12-01T10:40:42.7382917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03c16f_3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7383220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7383342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7383620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7383750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7383932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng4hunkn 2022-12-01T10:40:42.7384154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng4hunkn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7384422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7384540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7384815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7384944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7385126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssw3qvih 2022-12-01T10:40:42.7385322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssw3qvih/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7385588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7385700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7385971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7386103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7386284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphg96grwj 2022-12-01T10:40:42.7386477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphg96grwj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7386643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7386806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7386961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7387120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7387179Z ok (2.137s) 2022-12-01T10:40:42.7387185Z 2022-12-01T10:40:42.7387378Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7387451Z Ran 1 test in 2.138s 2022-12-01T10:40:42.7387456Z 2022-12-01T10:40:42.7387513Z OK 2022-12-01T10:40:42.7387518Z 2022-12-01T10:40:42.7387599Z Generating XML reports... 2022-12-01T10:40:42.7387970Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101451.xml 2022-12-01T10:40:42.7388238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7388360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7388627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7388759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7388943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqfd4g7l 2022-12-01T10:40:42.7389138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqfd4g7l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7389390Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7389395Z 2022-12-01T10:40:42.7389461Z Running tests... 2022-12-01T10:40:42.7389677Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7389920Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15548 2022-12-01T10:40:42.7390075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15549 2022-12-01T10:40:42.7390216Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15550 2022-12-01T10:40:42.7390396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15551 2022-12-01T10:40:42.7390672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7390792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7391074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7391209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7391392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrq5zzxk 2022-12-01T10:40:42.7391584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrq5zzxk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7391848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7391959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7392237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7392368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7392551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqo5pgpa 2022-12-01T10:40:42.7392744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqo5pgpa/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7393015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7393134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7393409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7393530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7393711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31c0_v_u 2022-12-01T10:40:42.7393896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31c0_v_u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7394162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7394281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7394560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7394690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7394872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdrwz_2y 2022-12-01T10:40:42.7395062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdrwz_2y/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7395215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7395377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7395535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7395696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7395899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7396076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7396376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7396545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7396705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7397034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7397332Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7397621Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7397903Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #4 to worker0, as the agent is shutting down 2022-12-01T10:40:42.7398294Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #5 to worker0, as the agent is shutting down 2022-12-01T10:40:42.7398560Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7398840Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #2 to worker1: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114) 2022-12-01T10:40:42.7399108Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7399382Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #3 to worker1: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114) 2022-12-01T10:40:42.7399644Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7399902Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7400148Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7400413Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7400671Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #6 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7400932Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7401195Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #4 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7401454Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #5 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7401759Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #7 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7402021Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7402275Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7402566Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #8 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7402818Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #9 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7403078Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7403344Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #6 to worker1: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114) 2022-12-01T10:40:42.7403599Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #7 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7403838Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114) 2022-12-01T10:40:42.7404050Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7404310Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7404381Z ok (2.137s) 2022-12-01T10:40:42.7404386Z 2022-12-01T10:40:42.7404593Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7404669Z Ran 1 test in 2.137s 2022-12-01T10:40:42.7404674Z 2022-12-01T10:40:42.7404739Z OK 2022-12-01T10:40:42.7404744Z 2022-12-01T10:40:42.7404831Z Generating XML reports... 2022-12-01T10:40:42.7405200Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101455.xml 2022-12-01T10:40:42.7405471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7405585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7405866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7406000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7406194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrugfchp 2022-12-01T10:40:42.7406394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrugfchp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7406650Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7406656Z 2022-12-01T10:40:42.7406726Z Running tests... 2022-12-01T10:40:42.7406927Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7407182Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15793 2022-12-01T10:40:42.7407356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15794 2022-12-01T10:40:42.7407516Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15795 2022-12-01T10:40:42.7407664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15796 2022-12-01T10:40:42.7407995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7408122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7408441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7408576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7408763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5htmc1cz 2022-12-01T10:40:42.7408948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5htmc1cz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7409219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7409339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7409617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7409746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7409931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05yv5zdd 2022-12-01T10:40:42.7410127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05yv5zdd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7410394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7410518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7410785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7410922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7411113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt1bsl0d 2022-12-01T10:40:42.7411384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7411510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7411791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7411924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7412099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshckwsr9 2022-12-01T10:40:42.7412282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt1bsl0d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7412473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshckwsr9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7412636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7412798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7412962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7413128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7413306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7413481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7413650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7413900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7414198Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7414493Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7414782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7415099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7415363Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7415659Z [W tensorpipe_agent.cpp:916] RPC agent for worker3 encountered error when sending outgoing request #4 to worker2: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:62) 2022-12-01T10:40:42.7415931Z [W tensorpipe_agent.cpp:679] RPC agent for worker3 encountered error when sending response to request #4 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7416188Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7416445Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7416707Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7416922Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7417171Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #8 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7417431Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #5 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7417687Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #10 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7417948Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #7 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7418205Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7418419Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7418680Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7418933Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7419214Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7419478Z [W tensorpipe_agent.cpp:940] RPC agent for worker1 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7419739Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #8 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7420023Z [W tensorpipe_agent.cpp:916] RPC agent for worker3 encountered error when sending outgoing request #6 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7420086Z ok (2.141s) 2022-12-01T10:40:42.7420091Z 2022-12-01T10:40:42.7420281Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7420354Z Ran 1 test in 2.141s 2022-12-01T10:40:42.7420359Z 2022-12-01T10:40:42.7420415Z OK 2022-12-01T10:40:42.7420423Z 2022-12-01T10:40:42.7420506Z Generating XML reports... 2022-12-01T10:40:42.7420877Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101458.xml 2022-12-01T10:40:42.7421149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7421273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7421559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7421696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7421872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8son7js 2022-12-01T10:40:42.7422068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8son7js/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7422324Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7422330Z 2022-12-01T10:40:42.7422401Z Running tests... 2022-12-01T10:40:42.7422591Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7422835Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16038 2022-12-01T10:40:42.7422989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16039 2022-12-01T10:40:42.7423145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16040 2022-12-01T10:40:42.7423286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16041 2022-12-01T10:40:42.7423559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7423680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7423962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7424097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7424281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbn7kzcga 2022-12-01T10:40:42.7424479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbn7kzcga/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7424748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7424865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7425131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7425264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7425483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzixli4f 2022-12-01T10:40:42.7425680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzixli4f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7425950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7426069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7426372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7426507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7426679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphx56n6_d 2022-12-01T10:40:42.7426872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphx56n6_d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7427137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7427260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7427533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7427667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7427847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahblyam5 2022-12-01T10:40:42.7428040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahblyam5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7428201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7428352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7428512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7428675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7428743Z ok (2.139s) 2022-12-01T10:40:42.7428748Z 2022-12-01T10:40:42.7428945Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7429019Z Ran 1 test in 2.140s 2022-12-01T10:40:42.7429024Z 2022-12-01T10:40:42.7429081Z OK 2022-12-01T10:40:42.7429085Z 2022-12-01T10:40:42.7429166Z Generating XML reports... 2022-12-01T10:40:42.7429526Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101502.xml 2022-12-01T10:40:42.7429795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7429917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7430198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7430332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7430512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_b0mfm6g 2022-12-01T10:40:42.7430699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_b0mfm6g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7430952Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7430960Z 2022-12-01T10:40:42.7431031Z Running tests... 2022-12-01T10:40:42.7431213Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7431447Z test_backward_rref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16281 2022-12-01T10:40:42.7431602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16282 2022-12-01T10:40:42.7431780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16283 2022-12-01T10:40:42.7431934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16284 2022-12-01T10:40:42.7432208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7432328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7432608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7432769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7432943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52unw4vp 2022-12-01T10:40:42.7433133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52unw4vp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7433405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7433526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7433803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7433935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7434115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3y2z_k7z 2022-12-01T10:40:42.7434307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3y2z_k7z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7434565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7434687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7434962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7435094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7435275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91xtqegn 2022-12-01T10:40:42.7435470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91xtqegn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7435738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7435857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7436135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7436256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7436434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxk32f41 2022-12-01T10:40:42.7436626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxk32f41/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7436787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7436949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7437108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7437267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7437334Z ok (2.142s) 2022-12-01T10:40:42.7437340Z 2022-12-01T10:40:42.7437523Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7437591Z Ran 1 test in 2.142s 2022-12-01T10:40:42.7437596Z 2022-12-01T10:40:42.7437654Z OK 2022-12-01T10:40:42.7437659Z 2022-12-01T10:40:42.7437742Z Generating XML reports... 2022-12-01T10:40:42.7438136Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101506.xml 2022-12-01T10:40:42.7438520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7438637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7438916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7439049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7439271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10tbnjdk 2022-12-01T10:40:42.7439458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10tbnjdk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7439712Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7439717Z 2022-12-01T10:40:42.7439787Z Running tests... 2022-12-01T10:40:42.7439982Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7440224Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16524 2022-12-01T10:40:42.7440379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16525 2022-12-01T10:40:42.7440533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16526 2022-12-01T10:40:42.7440677Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16527 2022-12-01T10:40:42.7440949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7441070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7441345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7441476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7441654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91sqthk7 2022-12-01T10:40:42.7441844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91sqthk7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7442105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7442221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7442489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7442616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7442793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpix1u2wct 2022-12-01T10:40:42.7442981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpix1u2wct/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7443247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7443363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7443637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7443764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7443938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnt30hhjv 2022-12-01T10:40:42.7444126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnt30hhjv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7444389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7444504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7444812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7444941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7445117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdudt2oui 2022-12-01T10:40:42.7445308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdudt2oui/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7445493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7445644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7445800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7445957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7446021Z ok (2.037s) 2022-12-01T10:40:42.7446026Z 2022-12-01T10:40:42.7453992Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7454070Z Ran 1 test in 2.037s 2022-12-01T10:40:42.7454076Z 2022-12-01T10:40:42.7454130Z OK 2022-12-01T10:40:42.7454135Z 2022-12-01T10:40:42.7454218Z Generating XML reports... 2022-12-01T10:40:42.7454599Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101509.xml 2022-12-01T10:40:42.7454863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7454991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7455272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7455409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7455599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3qpjrcf 2022-12-01T10:40:42.7455796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3qpjrcf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7456053Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7456059Z 2022-12-01T10:40:42.7456133Z Running tests... 2022-12-01T10:40:42.7456318Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7456565Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16767 2022-12-01T10:40:42.7456722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16768 2022-12-01T10:40:42.7456878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16769 2022-12-01T10:40:42.7457033Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16770 2022-12-01T10:40:42.7457311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7457436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7457716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7457853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7458026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8dxwwre 2022-12-01T10:40:42.7458225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8dxwwre/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7458496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7458619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7458975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7459111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7459294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphm_686_y 2022-12-01T10:40:42.7459485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphm_686_y/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7459757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7459902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7460184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7460318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7460506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg19qdjz1 2022-12-01T10:40:42.7460702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg19qdjz1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7460971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7461093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7461372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7461495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7461676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkm_azcf5 2022-12-01T10:40:42.7461869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkm_azcf5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7462033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7462200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7462363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7462524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7462594Z ok (2.240s) 2022-12-01T10:40:42.7462599Z 2022-12-01T10:40:42.7462801Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7462868Z Ran 1 test in 2.240s 2022-12-01T10:40:42.7462873Z 2022-12-01T10:40:42.7462933Z OK 2022-12-01T10:40:42.7462937Z 2022-12-01T10:40:42.7463021Z Generating XML reports... 2022-12-01T10:40:42.7463389Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101513.xml 2022-12-01T10:40:42.7463661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7463787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7464066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7464202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7464372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqf4t0no 2022-12-01T10:40:42.7464567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqf4t0no/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7464826Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7464831Z 2022-12-01T10:40:42.7464904Z Running tests... 2022-12-01T10:40:42.7465100Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7465339Z test_backward_simple (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17010 2022-12-01T10:40:42.7465534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17011 2022-12-01T10:40:42.7465692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17012 2022-12-01T10:40:42.7465847Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17013 2022-12-01T10:40:42.7466112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7466263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7466545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7466682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7466868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yqi24yx 2022-12-01T10:40:42.7467066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yqi24yx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7467341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7467464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7467733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7467867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7468055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyxg28db 2022-12-01T10:40:42.7468252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyxg28db/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7468525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7468650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7468932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7469069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7469252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqckxvxs 2022-12-01T10:40:42.7469434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqckxvxs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7469708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7469832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7470112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7470247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7470433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcv65uyp 2022-12-01T10:40:42.7470628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcv65uyp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7470793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7470957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7471110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7471272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7471341Z ok (2.138s) 2022-12-01T10:40:42.7471345Z 2022-12-01T10:40:42.7471543Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7471618Z Ran 1 test in 2.138s 2022-12-01T10:40:42.7471622Z 2022-12-01T10:40:42.7471684Z OK 2022-12-01T10:40:42.7471689Z 2022-12-01T10:40:42.7471801Z Generating XML reports... 2022-12-01T10:40:42.7472173Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101516.xml 2022-12-01T10:40:42.7472433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7472556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7472884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7473019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7473206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsed4k44l 2022-12-01T10:40:42.7473401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsed4k44l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7473660Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7473665Z 2022-12-01T10:40:42.7473739Z Running tests... 2022-12-01T10:40:42.7473935Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7474172Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17253 2022-12-01T10:40:42.7474330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17254 2022-12-01T10:40:42.7474487Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17255 2022-12-01T10:40:42.7474642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17256 2022-12-01T10:40:42.7474918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7475041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7475327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7475465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7475636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bjajm1n 2022-12-01T10:40:42.7475831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bjajm1n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7476104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7476229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7476507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7476642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7476831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1hjk4_0 2022-12-01T10:40:42.7477026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1hjk4_0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7477294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7477404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7477682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7477819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7478003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbg88vai5 2022-12-01T10:40:42.7478317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbg88vai5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7478642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7478765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7479044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7479165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7479349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5k_ber0 2022-12-01T10:40:42.7479577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5k_ber0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7479742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7479907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7480069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7480235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7480306Z ok (2.039s) 2022-12-01T10:40:42.7480311Z 2022-12-01T10:40:42.7480509Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7480572Z Ran 1 test in 2.039s 2022-12-01T10:40:42.7480577Z 2022-12-01T10:40:42.7480639Z OK 2022-12-01T10:40:42.7480643Z 2022-12-01T10:40:42.7480728Z Generating XML reports... 2022-12-01T10:40:42.7481101Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101520.xml 2022-12-01T10:40:42.7481372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7481495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7481775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7481912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7482082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4i68o01y 2022-12-01T10:40:42.7482273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4i68o01y/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7482527Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7482534Z 2022-12-01T10:40:42.7482608Z Running tests... 2022-12-01T10:40:42.7482804Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7483055Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17496 2022-12-01T10:40:42.7483214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17497 2022-12-01T10:40:42.7483373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17498 2022-12-01T10:40:42.7483529Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17499 2022-12-01T10:40:42.7483787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7483911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7484189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7484326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7484511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrgqx5va 2022-12-01T10:40:42.7484707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrgqx5va/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7484977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7485130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7485410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7485533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7485718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56hw4n_5 2022-12-01T10:40:42.7485941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56hw4n_5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7486209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7486331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7486608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7486745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7486931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5btb5x6 2022-12-01T10:40:42.7487110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5btb5x6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7487378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7487499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7487837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7487975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7488160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwjy7428 2022-12-01T10:40:42.7488352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwjy7428/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7488519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7488685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7488834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7489000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7489071Z ok (2.238s) 2022-12-01T10:40:42.7489076Z 2022-12-01T10:40:42.7489278Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7489355Z Ran 1 test in 2.238s 2022-12-01T10:40:42.7489360Z 2022-12-01T10:40:42.7489422Z OK 2022-12-01T10:40:42.7489427Z 2022-12-01T10:40:42.7489513Z Generating XML reports... 2022-12-01T10:40:42.7489882Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101523.xml 2022-12-01T10:40:42.7490148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7490275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7490556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7490695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7490885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmstpid2 2022-12-01T10:40:42.7491085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmstpid2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7491341Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7491346Z 2022-12-01T10:40:42.7491421Z Running tests... 2022-12-01T10:40:42.7491650Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7491881Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17739 2022-12-01T10:40:42.7492042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17740 2022-12-01T10:40:42.7492198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17741 2022-12-01T10:40:42.7492352Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17742 2022-12-01T10:40:42.7492658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7492781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7493062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7493198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7493373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfw0jzpnv 2022-12-01T10:40:42.7493569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfw0jzpnv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7493839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7493963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7494244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7494378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7494567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7s9ve9s 2022-12-01T10:40:42.7494762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7s9ve9s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7495034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7495145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7495422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7495556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7495740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfatz_4yp 2022-12-01T10:40:42.7495935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfatz_4yp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7496202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7496325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7496605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7496739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7496909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjejcwk0x 2022-12-01T10:40:42.7497107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjejcwk0x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7497271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7497438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7497601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7497764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7497833Z ok (2.140s) 2022-12-01T10:40:42.7497838Z 2022-12-01T10:40:42.7498070Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7498135Z Ran 1 test in 2.140s 2022-12-01T10:40:42.7498140Z 2022-12-01T10:40:42.7498200Z OK 2022-12-01T10:40:42.7498204Z 2022-12-01T10:40:42.7498289Z Generating XML reports... 2022-12-01T10:40:42.7498662Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101527.xml 2022-12-01T10:40:42.7498936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7499087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7499366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7499501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7499673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgpcceib 2022-12-01T10:40:42.7499872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgpcceib/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7500129Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7500134Z 2022-12-01T10:40:42.7500209Z Running tests... 2022-12-01T10:40:42.7500405Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7500655Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17982 2022-12-01T10:40:42.7500815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17983 2022-12-01T10:40:42.7500970Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17984 2022-12-01T10:40:42.7501123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17985 2022-12-01T10:40:42.7501385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7501511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7501794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7501930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7502114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_wo8i97 2022-12-01T10:40:42.7502311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_wo8i97/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7502587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7502711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7502993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7503117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7503302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf169p57e 2022-12-01T10:40:42.7503496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf169p57e/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7503766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7503891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7504171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7504308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7504492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98ycen4t 2022-12-01T10:40:42.7504697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98ycen4t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7504971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7505096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7505376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7505542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7505726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6p0unkb 2022-12-01T10:40:42.7505922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6p0unkb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7506086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7506252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7506402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7506563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7506631Z ok (12.153s) 2022-12-01T10:40:42.7506636Z 2022-12-01T10:40:42.7506835Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7506916Z Ran 1 test in 12.154s 2022-12-01T10:40:42.7506921Z 2022-12-01T10:40:42.7506982Z OK 2022-12-01T10:40:42.7506987Z 2022-12-01T10:40:42.7507070Z Generating XML reports... 2022-12-01T10:40:42.7507444Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101531.xml 2022-12-01T10:40:42.7507704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7507834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7508115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7508250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7508436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvs70q21 2022-12-01T10:40:42.7508629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvs70q21/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7508886Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7508892Z 2022-12-01T10:40:42.7508965Z Running tests... 2022-12-01T10:40:42.7509164Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7509857Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77556 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.221s) 2022-12-01T10:40:42.7509876Z 2022-12-01T10:40:42.7510058Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7510134Z Ran 1 test in 0.222s 2022-12-01T10:40:42.7510139Z 2022-12-01T10:40:42.7510211Z OK (skipped=1) 2022-12-01T10:40:42.7510218Z 2022-12-01T10:40:42.7510304Z Generating XML reports... 2022-12-01T10:40:42.7510672Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101544.xml 2022-12-01T10:40:42.7510947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7511071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7511394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7511532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7511707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynmxp9_4 2022-12-01T10:40:42.7511907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynmxp9_4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7512164Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7512196Z 2022-12-01T10:40:42.7512272Z Running tests... 2022-12-01T10:40:42.7512469Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7512713Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18239 2022-12-01T10:40:42.7512870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18240 2022-12-01T10:40:42.7513029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18241 2022-12-01T10:40:42.7513184Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18242 2022-12-01T10:40:42.7513445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7513570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7513853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7513993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7514179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpanrd7j7v 2022-12-01T10:40:42.7514377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpanrd7j7v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7514651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7514778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7515046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7515181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7515367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptazo28th 2022-12-01T10:40:42.7515566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptazo28th/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7515837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7515960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7516242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7516376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7516562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwdxkkak 2022-12-01T10:40:42.7516745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwdxkkak/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7517019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7517145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7517426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7517561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7517744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp894oec1w 2022-12-01T10:40:42.7517966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp894oec1w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7518131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7518407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7518569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7518774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7518844Z ok (2.239s) 2022-12-01T10:40:42.7518850Z 2022-12-01T10:40:42.7519051Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7519130Z Ran 1 test in 2.239s 2022-12-01T10:40:42.7519135Z 2022-12-01T10:40:42.7519196Z OK 2022-12-01T10:40:42.7519201Z 2022-12-01T10:40:42.7519287Z Generating XML reports... 2022-12-01T10:40:42.7519660Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101546.xml 2022-12-01T10:40:42.7519922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7520046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7520326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7520464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7520651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sou6m31 2022-12-01T10:40:42.7520848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sou6m31/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7521105Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7521110Z 2022-12-01T10:40:42.7521188Z Running tests... 2022-12-01T10:40:42.7521372Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7521621Z test_backward_without_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18482 2022-12-01T10:40:42.7521779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18483 2022-12-01T10:40:42.7521935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18484 2022-12-01T10:40:42.7522091Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18485 2022-12-01T10:40:42.7522366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7522492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7522775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7522914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7523087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcew5velg 2022-12-01T10:40:42.7523284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcew5velg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7523555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7523680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7523960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7524094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7524278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_fdsj0n 2022-12-01T10:40:42.7524506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_fdsj0n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7524779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7524889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7525165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7525330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7525513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozdnqe74 2022-12-01T10:40:42.7525707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozdnqe74/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7525978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7526104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7526380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7526501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7526684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt348_qp 2022-12-01T10:40:42.7526879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt348_qp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7527048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7527214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7527375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7527539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7527610Z ok (2.139s) 2022-12-01T10:40:42.7527617Z 2022-12-01T10:40:42.7527874Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7527938Z Ran 1 test in 2.139s 2022-12-01T10:40:42.7527943Z 2022-12-01T10:40:42.7528004Z OK 2022-12-01T10:40:42.7528009Z 2022-12-01T10:40:42.7528094Z Generating XML reports... 2022-12-01T10:40:42.7528465Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101550.xml 2022-12-01T10:40:42.7528741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7528864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7529145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7529280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7529452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxastkxrv 2022-12-01T10:40:42.7529650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxastkxrv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7529904Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7529909Z 2022-12-01T10:40:42.7529983Z Running tests... 2022-12-01T10:40:42.7530178Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7530423Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18709 2022-12-01T10:40:42.7530583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18710 2022-12-01T10:40:42.7530741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18711 2022-12-01T10:40:42.7530893Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18712 2022-12-01T10:40:42.7531188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7531312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7531593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7531728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7531942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7yxd2iv 2022-12-01T10:40:42.7532140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7yxd2iv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7532413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7532537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7532808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7532944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7533129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvqm9a7z 2022-12-01T10:40:42.7533326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvqm9a7z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7533599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7533720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7534000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7534137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7534324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby8x8lf6 2022-12-01T10:40:42.7534506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby8x8lf6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7534779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7534903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7535184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7535322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7535507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9qnat8w 2022-12-01T10:40:42.7535701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9qnat8w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7535869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7536020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7536183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7536344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7536412Z ok (2.038s) 2022-12-01T10:40:42.7536417Z 2022-12-01T10:40:42.7536615Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7536693Z Ran 1 test in 2.038s 2022-12-01T10:40:42.7536698Z 2022-12-01T10:40:42.7536758Z OK 2022-12-01T10:40:42.7536763Z 2022-12-01T10:40:42.7536849Z Generating XML reports... 2022-12-01T10:40:42.7537221Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101553.xml 2022-12-01T10:40:42.7537512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7537638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7537917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7538053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7538237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo__o6wsi 2022-12-01T10:40:42.7538465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo__o6wsi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7538720Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7538726Z 2022-12-01T10:40:42.7538800Z Running tests... 2022-12-01T10:40:42.7538984Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7539235Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18952 2022-12-01T10:40:42.7539393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18953 2022-12-01T10:40:42.7539548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18954 2022-12-01T10:40:42.7539704Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18955 2022-12-01T10:40:42.7539978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7540103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7540383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7540521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7540693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1osh847 2022-12-01T10:40:42.7540890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1osh847/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7541167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7541290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7541568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7541706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7541892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph86j8myu 2022-12-01T10:40:42.7542089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph86j8myu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7542359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7542473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7542751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7542886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7543070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy57xapr5 2022-12-01T10:40:42.7543264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy57xapr5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7543536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7543660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7543938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7544087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7544273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppr0ia92w 2022-12-01T10:40:42.7544467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppr0ia92w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7544633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7544798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7544987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7545149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7545218Z ok (2.141s) 2022-12-01T10:40:42.7545222Z 2022-12-01T10:40:42.7545421Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7545484Z Ran 1 test in 2.141s 2022-12-01T10:40:42.7545489Z 2022-12-01T10:40:42.7545553Z OK 2022-12-01T10:40:42.7545558Z 2022-12-01T10:40:42.7545641Z Generating XML reports... 2022-12-01T10:40:42.7546013Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101557.xml 2022-12-01T10:40:42.7546283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7546409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7546692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7546828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7547001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97hpgejc 2022-12-01T10:40:42.7547196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97hpgejc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7547454Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7547459Z 2022-12-01T10:40:42.7547533Z Running tests... 2022-12-01T10:40:42.7547728Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7547884Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.7548228Z This test simulates the situation where the 'backward' call might throw ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19195 2022-12-01T10:40:42.7548389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19196 2022-12-01T10:40:42.7548545Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19197 2022-12-01T10:40:42.7548684Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19198 2022-12-01T10:40:42.7548961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7549085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7549365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7549501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7549688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bxeeq29 2022-12-01T10:40:42.7549886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bxeeq29/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7550162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7550272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7550553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7550716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7550906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwudwi31 2022-12-01T10:40:42.7551101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwudwi31/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7551373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7551530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7551811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7551946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7552118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvqkbte4 2022-12-01T10:40:42.7552316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvqkbte4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7552588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7552712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7552992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7553129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7553314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumf629_m 2022-12-01T10:40:42.7553507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumf629_m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7553675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7553830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7553992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7554154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7554331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7554510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7554686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7554988Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7555163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7555450Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7555749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7556039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7556312Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7556584Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7556849Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7557145Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7557405Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7557669Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7557949Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7558364Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7558621Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7558878Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7559119Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7559375Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7559446Z ok (2.342s) 2022-12-01T10:40:42.7559452Z 2022-12-01T10:40:42.7559657Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7559734Z Ran 1 test in 2.342s 2022-12-01T10:40:42.7559739Z 2022-12-01T10:40:42.7559801Z OK 2022-12-01T10:40:42.7559806Z 2022-12-01T10:40:42.7559892Z Generating XML reports... 2022-12-01T10:40:42.7560264Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101600.xml 2022-12-01T10:40:42.7560543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7560657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7560941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7561078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7561266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqo1uhqa 2022-12-01T10:40:42.7561467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqo1uhqa/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7561723Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7561728Z 2022-12-01T10:40:42.7561802Z Running tests... 2022-12-01T10:40:42.7562000Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7562250Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19434 2022-12-01T10:40:42.7562399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19435 2022-12-01T10:40:42.7562554Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19436 2022-12-01T10:40:42.7562708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19437 2022-12-01T10:40:42.7563029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7563151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7563433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7563570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7563756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48n9kjp1 2022-12-01T10:40:42.7563989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48n9kjp1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7564248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7564371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7564653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7564788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7564976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gxkaf73 2022-12-01T10:40:42.7565172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gxkaf73/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7565443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7565570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7565835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7565972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7566157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8pdhegb 2022-12-01T10:40:42.7566357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8pdhegb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7566626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7566750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7567027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7567164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7567349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzi67zm8 2022-12-01T10:40:42.7567528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzi67zm8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7567746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7567919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7568083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7568247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7568428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7568606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7568783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7568942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7569248Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7569584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7569877Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7570170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7570238Z ok (2.238s) 2022-12-01T10:40:42.7570244Z 2022-12-01T10:40:42.7570469Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7570546Z Ran 1 test in 2.239s 2022-12-01T10:40:42.7570552Z 2022-12-01T10:40:42.7570613Z OK 2022-12-01T10:40:42.7570617Z 2022-12-01T10:40:42.7570690Z Generating XML reports... 2022-12-01T10:40:42.7571061Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101604.xml 2022-12-01T10:40:42.7571339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7571464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7571746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7571881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7572069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62ios6h0 2022-12-01T10:40:42.7572271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62ios6h0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7572531Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7572536Z 2022-12-01T10:40:42.7572598Z Running tests... 2022-12-01T10:40:42.7572796Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7573046Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19669 2022-12-01T10:40:42.7573207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19670 2022-12-01T10:40:42.7573367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19671 2022-12-01T10:40:42.7573523Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19672 2022-12-01T10:40:42.7573804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7573929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7574213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7574335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7574526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjvz31lk 2022-12-01T10:40:42.7574724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjvz31lk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7574998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7575123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7575406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7575546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7575732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfg7mjom3 2022-12-01T10:40:42.7575917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfg7mjom3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7576248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7576371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7576655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7576792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7576977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1d03xjo 2022-12-01T10:40:42.7577203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1d03xjo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7577472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7577596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7577862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7578000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7578184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbm69wuii 2022-12-01T10:40:42.7578379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbm69wuii/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7578543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7578710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7578874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7579036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7579200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7579378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7579555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7579728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7580033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7580334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7580626Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7580917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7580984Z ok (2.237s) 2022-12-01T10:40:42.7580989Z 2022-12-01T10:40:42.7581189Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7581253Z Ran 1 test in 2.237s 2022-12-01T10:40:42.7581258Z 2022-12-01T10:40:42.7581318Z OK 2022-12-01T10:40:42.7581323Z 2022-12-01T10:40:42.7581407Z Generating XML reports... 2022-12-01T10:40:42.7581778Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101608.xml 2022-12-01T10:40:42.7582053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7582179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7582459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7582595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7582773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz7l8i2s 2022-12-01T10:40:42.7582999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz7l8i2s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7583255Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7583260Z 2022-12-01T10:40:42.7583336Z Running tests... 2022-12-01T10:40:42.7583531Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7583784Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19904 2022-12-01T10:40:42.7583969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19905 2022-12-01T10:40:42.7584128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19906 2022-12-01T10:40:42.7584282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19907 2022-12-01T10:40:42.7584547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7584671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7584952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7585088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7585274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_mej3f4 2022-12-01T10:40:42.7585471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_mej3f4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7585743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7585866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7586133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7586268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7586453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvn29m_t 2022-12-01T10:40:42.7586652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvn29m_t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7586923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7587047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7587326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7587461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7587646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lvnge_p 2022-12-01T10:40:42.7587832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lvnge_p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7588103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7588225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7588505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7588644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7588827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5botngwe 2022-12-01T10:40:42.7589024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5botngwe/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7589191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7589370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7589536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7589699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7589878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7590055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7590256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7590430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7590729Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7591031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7591311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7591601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7591671Z ok (2.338s) 2022-12-01T10:40:42.7591676Z 2022-12-01T10:40:42.7591875Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7591951Z Ran 1 test in 2.338s 2022-12-01T10:40:42.7591957Z 2022-12-01T10:40:42.7592017Z OK 2022-12-01T10:40:42.7592022Z 2022-12-01T10:40:42.7592106Z Generating XML reports... 2022-12-01T10:40:42.7592480Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101611.xml 2022-12-01T10:40:42.7592757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7592869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7593153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7593289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7593475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1tllton 2022-12-01T10:40:42.7593675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1tllton/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7593930Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7593936Z 2022-12-01T10:40:42.7594010Z Running tests... 2022-12-01T10:40:42.7594208Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7594466Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20139 2022-12-01T10:40:42.7594613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20140 2022-12-01T10:40:42.7594771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20141 2022-12-01T10:40:42.7594927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20142 2022-12-01T10:40:42.7595203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7595326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7595606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7595741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7595954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5m8o44xc 2022-12-01T10:40:42.7596139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5m8o44xc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7596412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7596536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7596814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7596980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7597165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7aqq0xx4 2022-12-01T10:40:42.7597359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7aqq0xx4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7597631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7597752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7598015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7598148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7598469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_rk4t2q 2022-12-01T10:40:42.7598665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_rk4t2q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7598936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7599059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7599338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7599478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7599664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzao9b4jh 2022-12-01T10:40:42.7599847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzao9b4jh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7600012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7600181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7600346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7600507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7600686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7600869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7601046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7601208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7601510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7601810Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7602102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7602393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7602462Z ok (2.339s) 2022-12-01T10:40:42.7602467Z 2022-12-01T10:40:42.7602732Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7602809Z Ran 1 test in 2.339s 2022-12-01T10:40:42.7602814Z 2022-12-01T10:40:42.7602876Z OK 2022-12-01T10:40:42.7602881Z 2022-12-01T10:40:42.7602952Z Generating XML reports... 2022-12-01T10:40:42.7603325Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101615.xml 2022-12-01T10:40:42.7603602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7603764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7604045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7604184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7604374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcejn82he 2022-12-01T10:40:42.7604573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcejn82he/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7604828Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7604834Z 2022-12-01T10:40:42.7604893Z Running tests... 2022-12-01T10:40:42.7605089Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7605323Z test_debug_info (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20374 2022-12-01T10:40:42.7605480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20375 2022-12-01T10:40:42.7605637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20376 2022-12-01T10:40:42.7605792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20377 2022-12-01T10:40:42.7606072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7606197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7606464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7606602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7606787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmykmvrs 2022-12-01T10:40:42.7606986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmykmvrs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7607258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7607381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7607668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7607865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7608051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8w1gzckf 2022-12-01T10:40:42.7608232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8w1gzckf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7608502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7608629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7608907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7609042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7609227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8m6i6oh7 2022-12-01T10:40:42.7609455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8m6i6oh7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7609726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7609850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7610117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7610281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7610466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0nl6vsr 2022-12-01T10:40:42.7610663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0nl6vsr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7610828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7610995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7611159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7611321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7611486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7611664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7611842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7612015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7612319Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7612620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7612915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7613211Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7613281Z ok (2.340s) 2022-12-01T10:40:42.7613286Z 2022-12-01T10:40:42.7613473Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7613548Z Ran 1 test in 2.341s 2022-12-01T10:40:42.7613553Z 2022-12-01T10:40:42.7613614Z OK 2022-12-01T10:40:42.7613618Z 2022-12-01T10:40:42.7613703Z Generating XML reports... 2022-12-01T10:40:42.7614073Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101619.xml 2022-12-01T10:40:42.7614350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7614475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7614756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7614893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7615067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j2hamiq 2022-12-01T10:40:42.7615268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j2hamiq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7615526Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7615531Z 2022-12-01T10:40:42.7615606Z Running tests... 2022-12-01T10:40:42.7615804Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7616542Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77318 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.219s) 2022-12-01T10:40:42.7616549Z 2022-12-01T10:40:42.7616746Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7616850Z Ran 1 test in 0.219s 2022-12-01T10:40:42.7616855Z 2022-12-01T10:40:42.7616929Z OK (skipped=1) 2022-12-01T10:40:42.7616934Z 2022-12-01T10:40:42.7617020Z Generating XML reports... 2022-12-01T10:40:42.7617375Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101623.xml 2022-12-01T10:40:42.7617651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7617778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7618061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7618198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7618387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0dxc0x_ 2022-12-01T10:40:42.7618586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0dxc0x_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7618843Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7618848Z 2022-12-01T10:40:42.7618922Z Running tests... 2022-12-01T10:40:42.7619105Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7619340Z test_error_in_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20639 2022-12-01T10:40:42.7619501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20640 2022-12-01T10:40:42.7619657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20641 2022-12-01T10:40:42.7619814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20642 2022-12-01T10:40:42.7620088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7620215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7620496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7620619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7620808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbgf1sw3 2022-12-01T10:40:42.7621009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbgf1sw3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7621281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7621404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7621687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7621821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7622007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7vmgb8k 2022-12-01T10:40:42.7622199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7vmgb8k/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7622458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7622581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7622890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7623030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7623215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp1r2xw7 2022-12-01T10:40:42.7623412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp1r2xw7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7623710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7623835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7624103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7624237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7624421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnt6ve1zx 2022-12-01T10:40:42.7624617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnt6ve1zx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7624781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7624947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7625114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7625276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7625331Z ok (2.139s) 2022-12-01T10:40:42.7625350Z 2022-12-01T10:40:42.7625536Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7625614Z Ran 1 test in 2.139s 2022-12-01T10:40:42.7625619Z 2022-12-01T10:40:42.7625679Z OK 2022-12-01T10:40:42.7625684Z 2022-12-01T10:40:42.7625772Z Generating XML reports... 2022-12-01T10:40:42.7626142Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101624.xml 2022-12-01T10:40:42.7626417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7626542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7626827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7626949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7627136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsovwig6 2022-12-01T10:40:42.7627335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsovwig6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7627595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7627601Z 2022-12-01T10:40:42.7627674Z Running tests... 2022-12-01T10:40:42.7627869Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7628124Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20862 2022-12-01T10:40:42.7628282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20863 2022-12-01T10:40:42.7628443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20864 2022-12-01T10:40:42.7628584Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20865 2022-12-01T10:40:42.7628859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7628983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7629288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7629425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7629612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmg768n5 2022-12-01T10:40:42.7629808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmg768n5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7630111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7630221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7630503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7630639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7630827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4loezng 2022-12-01T10:40:42.7631023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4loezng/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7631295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7631419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7631698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7631837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7632009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6j8dspi 2022-12-01T10:40:42.7632204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6j8dspi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7632479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7632603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7632883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7633018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7633203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqs6wrpsn 2022-12-01T10:40:42.7633401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqs6wrpsn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7633565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7633716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7633879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7634043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7634533Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-12-01T10:40:42.7634674Z "Argument order of nn.functional.embedding_bag was changed. " 2022-12-01T10:40:42.7635166Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-12-01T10:40:42.7635306Z "Argument order of nn.functional.embedding_bag was changed. " 2022-12-01T10:40:42.7635821Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-12-01T10:40:42.7635962Z "Argument order of nn.functional.embedding_bag was changed. " 2022-12-01T10:40:42.7636446Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-12-01T10:40:42.7636607Z "Argument order of nn.functional.embedding_bag was changed. " 2022-12-01T10:40:42.7636676Z ok (2.038s) 2022-12-01T10:40:42.7636681Z 2022-12-01T10:40:42.7636882Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7636958Z Ran 1 test in 2.038s 2022-12-01T10:40:42.7636963Z 2022-12-01T10:40:42.7637024Z OK 2022-12-01T10:40:42.7637029Z 2022-12-01T10:40:42.7637115Z Generating XML reports... 2022-12-01T10:40:42.7637482Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101628.xml 2022-12-01T10:40:42.7637756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7637881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7638148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7638407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7638597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaybt_zxv 2022-12-01T10:40:42.7638796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaybt_zxv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7639053Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7639058Z 2022-12-01T10:40:42.7639133Z Running tests... 2022-12-01T10:40:42.7639332Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7639579Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21117 2022-12-01T10:40:42.7639723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21118 2022-12-01T10:40:42.7639882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21119 2022-12-01T10:40:42.7640038Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21120 2022-12-01T10:40:42.7640312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7640439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7640723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7640861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7641050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_xg0s2p 2022-12-01T10:40:42.7641246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_xg0s2p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7641506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7641634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7641916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7642051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7642236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywfb7wgq 2022-12-01T10:40:42.7642476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywfb7wgq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7642750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7642873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7643139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7643274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7643496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvycns86 2022-12-01T10:40:42.7643693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvycns86/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7643964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7644088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7644368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7644503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7644687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__j_miyp 2022-12-01T10:40:42.7644865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__j_miyp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7645033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7645199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7645362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7645527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7645708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7645887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7646063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7646365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7646528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7646824Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7647124Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7647417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7647486Z ok (2.338s) 2022-12-01T10:40:42.7647490Z 2022-12-01T10:40:42.7647741Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7647822Z Ran 1 test in 2.338s 2022-12-01T10:40:42.7647828Z 2022-12-01T10:40:42.7647891Z OK 2022-12-01T10:40:42.7647896Z 2022-12-01T10:40:42.7647982Z Generating XML reports... 2022-12-01T10:40:42.7648341Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101631.xml 2022-12-01T10:40:42.7648619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7648742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7649026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7649205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7649394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1eka0u02 2022-12-01T10:40:42.7649592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1eka0u02/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7649848Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7649853Z 2022-12-01T10:40:42.7649956Z Running tests... 2022-12-01T10:40:42.7650142Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7650394Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21372 2022-12-01T10:40:42.7650551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21373 2022-12-01T10:40:42.7650707Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21374 2022-12-01T10:40:42.7650863Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21375 2022-12-01T10:40:42.7651140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7651264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7651548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7651673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7651860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7d9sk_b_ 2022-12-01T10:40:42.7652055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7d9sk_b_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7652329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7652456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7652738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7652879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7653144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmput8k_rcv 2022-12-01T10:40:42.7653343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmput8k_rcv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7653605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7653729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7654009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7654145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7654330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0avchi3j 2022-12-01T10:40:42.7654530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0avchi3j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7654802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7654925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7655192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7655326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7655508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ndryh3w 2022-12-01T10:40:42.7655702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ndryh3w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7655911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7656077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7656240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7656402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7656610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7656775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7656949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7657124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7657425Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7657723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7658015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7658305Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7658379Z ok (2.339s) 2022-12-01T10:40:42.7658384Z 2022-12-01T10:40:42.7658581Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7658644Z Ran 1 test in 2.339s 2022-12-01T10:40:42.7658649Z 2022-12-01T10:40:42.7658713Z OK 2022-12-01T10:40:42.7658718Z 2022-12-01T10:40:42.7658803Z Generating XML reports... 2022-12-01T10:40:42.7659177Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101635.xml 2022-12-01T10:40:42.7659452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7659577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7659857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7659997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7660172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm17qzh3r 2022-12-01T10:40:42.7660371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm17qzh3r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7660625Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7660631Z 2022-12-01T10:40:42.7660707Z Running tests... 2022-12-01T10:40:42.7660905Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7661149Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21627 2022-12-01T10:40:42.7661311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21628 2022-12-01T10:40:42.7661471Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21629 2022-12-01T10:40:42.7661627Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21630 2022-12-01T10:40:42.7661887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7662010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7662290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7662455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7662645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjip2pre2 2022-12-01T10:40:42.7662842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjip2pre2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7663114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7663267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7663587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7663752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7663941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdfnht68_ 2022-12-01T10:40:42.7664139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdfnht68_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7664417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7664542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7664821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7664957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7665144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchi6dzwe 2022-12-01T10:40:42.7665326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchi6dzwe/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7665596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7665722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7666004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7666138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7666323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjat0x36 2022-12-01T10:40:42.7666517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjat0x36/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7666687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7666852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7667004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7667166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7667345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7667525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7667701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7667876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7668177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7668474Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7668767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7669078Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7669149Z ok (2.337s) 2022-12-01T10:40:42.7669154Z 2022-12-01T10:40:42.7669352Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7669430Z Ran 1 test in 2.338s 2022-12-01T10:40:42.7669436Z 2022-12-01T10:40:42.7669496Z OK 2022-12-01T10:40:42.7669501Z 2022-12-01T10:40:42.7669589Z Generating XML reports... 2022-12-01T10:40:42.7669962Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101639.xml 2022-12-01T10:40:42.7670267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7670379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7670661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7670799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7670988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8uq7oph 2022-12-01T10:40:42.7671188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8uq7oph/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7671442Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7671448Z 2022-12-01T10:40:42.7671523Z Running tests... 2022-12-01T10:40:42.7671719Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7671972Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21862 2022-12-01T10:40:42.7672118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21863 2022-12-01T10:40:42.7672274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21864 2022-12-01T10:40:42.7672431Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21865 2022-12-01T10:40:42.7672707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7672832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7673113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7673252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7673438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v12hpl5 2022-12-01T10:40:42.7673634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v12hpl5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7673891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7674016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7674298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7674433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7674622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcayath30 2022-12-01T10:40:42.7674824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcayath30/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7675098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7675221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7675489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7675650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7675834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpom1th1me 2022-12-01T10:40:42.7676031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpom1th1me/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7676300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7676424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7676753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7676889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7677074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25e0sdh2 2022-12-01T10:40:42.7677256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25e0sdh2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7677422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7677587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7677750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7677911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7678093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7678381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7678557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7678719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7679024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7679321Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7679617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7679908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7679979Z ok (2.340s) 2022-12-01T10:40:42.7679984Z 2022-12-01T10:40:42.7680185Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7680263Z Ran 1 test in 2.340s 2022-12-01T10:40:42.7680268Z 2022-12-01T10:40:42.7680330Z OK 2022-12-01T10:40:42.7680336Z 2022-12-01T10:40:42.7680408Z Generating XML reports... 2022-12-01T10:40:42.7680782Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101643.xml 2022-12-01T10:40:42.7681061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7681190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7681476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7681615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7681803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiu7s0qmq 2022-12-01T10:40:42.7682005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiu7s0qmq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7682261Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7682266Z 2022-12-01T10:40:42.7682327Z Running tests... 2022-12-01T10:40:42.7682572Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7682818Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22097 2022-12-01T10:40:42.7682976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22098 2022-12-01T10:40:42.7683131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22099 2022-12-01T10:40:42.7683323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22100 2022-12-01T10:40:42.7683598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7683726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7684008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7684133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7684320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiac1cs01 2022-12-01T10:40:42.7684518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiac1cs01/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7684791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7684919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7685200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7685339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7685523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53bj39k9 2022-12-01T10:40:42.7685707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53bj39k9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7685980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7686103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7686379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7686513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7686703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyzg7tdr 2022-12-01T10:40:42.7686902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyzg7tdr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7687174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7687299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7687567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7687754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7687947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8w5ptppe 2022-12-01T10:40:42.7688145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8w5ptppe/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7688312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7688479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7688643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7688806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7689005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7689183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7689360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7689535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7689866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7690168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7690460Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7690755Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7690824Z ok (2.241s) 2022-12-01T10:40:42.7690829Z 2022-12-01T10:40:42.7691028Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7691090Z Ran 1 test in 2.241s 2022-12-01T10:40:42.7691095Z 2022-12-01T10:40:42.7691156Z OK 2022-12-01T10:40:42.7691161Z 2022-12-01T10:40:42.7691246Z Generating XML reports... 2022-12-01T10:40:42.7691619Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101646.xml 2022-12-01T10:40:42.7691894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7692019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7692301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7692443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7692617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42t5lpk_ 2022-12-01T10:40:42.7692815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42t5lpk_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7693068Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7693073Z 2022-12-01T10:40:42.7693149Z Running tests... 2022-12-01T10:40:42.7693349Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7693598Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22332 2022-12-01T10:40:42.7693755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22333 2022-12-01T10:40:42.7693910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22334 2022-12-01T10:40:42.7694066Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22335 2022-12-01T10:40:42.7694329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7694452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7694734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7694874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7695060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpki1l9lyz 2022-12-01T10:40:42.7695258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpki1l9lyz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7695529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7695693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7695961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7696096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7696282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmhj2pih 2022-12-01T10:40:42.7696477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmhj2pih/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7696776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7696901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7697183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7697321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7697503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ddf_0eh 2022-12-01T10:40:42.7697684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ddf_0eh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7697957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7698080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7698360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7698495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7698681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3eln2i6 2022-12-01T10:40:42.7698875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3eln2i6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7699042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7699192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7699355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7699518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7699700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7699876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7700053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7700226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7700528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7700826Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7701107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7701398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7701470Z ok (2.238s) 2022-12-01T10:40:42.7701475Z 2022-12-01T10:40:42.7701671Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7701748Z Ran 1 test in 2.239s 2022-12-01T10:40:42.7701753Z 2022-12-01T10:40:42.7701817Z OK 2022-12-01T10:40:42.7701822Z 2022-12-01T10:40:42.7701907Z Generating XML reports... 2022-12-01T10:40:42.7702302Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101650.xml 2022-12-01T10:40:42.7702577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7702689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7702971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7703142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7703328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3wno89z 2022-12-01T10:40:42.7703525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3wno89z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7703780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7703786Z 2022-12-01T10:40:42.7703860Z Running tests... 2022-12-01T10:40:42.7704063Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7704317Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22567 2022-12-01T10:40:42.7704462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22568 2022-12-01T10:40:42.7704617Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22569 2022-12-01T10:40:42.7704774Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22570 2022-12-01T10:40:42.7705048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7705173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7705452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7705589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7705776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiar8wlcg 2022-12-01T10:40:42.7705960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiar8wlcg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7706233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7706360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7706638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7706775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7706959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_18x3iie 2022-12-01T10:40:42.7707155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_18x3iie/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7707429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7707552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7707817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7707950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7708138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5i242gh 2022-12-01T10:40:42.7708332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5i242gh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7708604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7708727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7709038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7709176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7709363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzbvxtu4 2022-12-01T10:40:42.7709544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzbvxtu4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7709738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7709903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7710065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7710228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7710406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7710582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7710760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7710919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7711220Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7711517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7711811Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7712106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7712175Z ok (2.243s) 2022-12-01T10:40:42.7712180Z 2022-12-01T10:40:42.7712378Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7712455Z Ran 1 test in 2.244s 2022-12-01T10:40:42.7712460Z 2022-12-01T10:40:42.7712522Z OK 2022-12-01T10:40:42.7712526Z 2022-12-01T10:40:42.7712599Z Generating XML reports... 2022-12-01T10:40:42.7712969Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101654.xml 2022-12-01T10:40:42.7713247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7713376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7713657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7713797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7713985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmclg5svb 2022-12-01T10:40:42.7714184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmclg5svb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7714440Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7714445Z 2022-12-01T10:40:42.7714507Z Running tests... 2022-12-01T10:40:42.7714703Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7714962Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22802 2022-12-01T10:40:42.7715121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22803 2022-12-01T10:40:42.7715274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22804 2022-12-01T10:40:42.7715456Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22805 2022-12-01T10:40:42.7715734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7715860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7716143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7716291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7716477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ik6y7_t 2022-12-01T10:40:42.7716674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ik6y7_t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7716945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7717070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7717353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7717488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7717673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8j_pfwi9 2022-12-01T10:40:42.7717854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8j_pfwi9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7718129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7718376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7718656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7718794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7718979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxh3cgy0 2022-12-01T10:40:42.7719173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxh3cgy0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7719444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7719567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7719838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7719975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7720160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpmuled6 2022-12-01T10:40:42.7720360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpmuled6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7720527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7720693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7720857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7721021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7721187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7721363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7721537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7721839Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7722182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7722358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7722652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7722950Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7723055Z ok (2.241s) 2022-12-01T10:40:42.7723060Z 2022-12-01T10:40:42.7723246Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7723323Z Ran 1 test in 2.241s 2022-12-01T10:40:42.7723328Z 2022-12-01T10:40:42.7723389Z OK 2022-12-01T10:40:42.7723394Z 2022-12-01T10:40:42.7723480Z Generating XML reports... 2022-12-01T10:40:42.7723853Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101657.xml 2022-12-01T10:40:42.7724129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7724254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7724540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7724679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7724852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp8qntu5 2022-12-01T10:40:42.7725127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp8qntu5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7725421Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7725427Z 2022-12-01T10:40:42.7725501Z Running tests... 2022-12-01T10:40:42.7725705Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7725949Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23037 2022-12-01T10:40:42.7726107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23038 2022-12-01T10:40:42.7726267Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23039 2022-12-01T10:40:42.7726426Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23040 2022-12-01T10:40:42.7726690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7726814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7727097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7727235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7727421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u023l4d 2022-12-01T10:40:42.7727618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u023l4d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7727947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7728078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7728348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7728486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7728671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpneij3itc 2022-12-01T10:40:42.7728918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpneij3itc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7729195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7729322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7729602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7729738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7729953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4m1ctqt 2022-12-01T10:40:42.7730132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4m1ctqt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7730406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7730534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7730815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7730950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7731137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wakeg3w 2022-12-01T10:40:42.7731332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wakeg3w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7731499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7731650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7731817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7731979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7732159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7732336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7732507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7732680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7732985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7733283Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7733562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7733857Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7733927Z ok (2.344s) 2022-12-01T10:40:42.7733932Z 2022-12-01T10:40:42.7734134Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7734210Z Ran 1 test in 2.344s 2022-12-01T10:40:42.7734215Z 2022-12-01T10:40:42.7734277Z OK 2022-12-01T10:40:42.7734281Z 2022-12-01T10:40:42.7734366Z Generating XML reports... 2022-12-01T10:40:42.7734737Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101701.xml 2022-12-01T10:40:42.7735013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7735125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7735409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7735578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7735765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ev1lcho 2022-12-01T10:40:42.7735965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ev1lcho/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7736222Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7736227Z 2022-12-01T10:40:42.7736328Z Running tests... 2022-12-01T10:40:42.7736525Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7736774Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23272 2022-12-01T10:40:42.7736920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23273 2022-12-01T10:40:42.7737075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23274 2022-12-01T10:40:42.7737231Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23275 2022-12-01T10:40:42.7737507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7737632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7737914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7738054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7738240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8m93r1xq 2022-12-01T10:40:42.7738422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8m93r1xq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7738693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7738819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7739099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7739234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7739420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprifj958z 2022-12-01T10:40:42.7739614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprifj958z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7739885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7740008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7740273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7740409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7740593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisg5j62t 2022-12-01T10:40:42.7740789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisg5j62t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7741058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7741181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7741465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7741601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7741772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe52sd7jt 2022-12-01T10:40:42.7741966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe52sd7jt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7742166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7742333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7742495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7742658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7742888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7743065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7743239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7743402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7743704Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7744004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7744299Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7744593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7744665Z ok (2.342s) 2022-12-01T10:40:42.7744670Z 2022-12-01T10:40:42.7744871Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7744947Z Ran 1 test in 2.342s 2022-12-01T10:40:42.7744952Z 2022-12-01T10:40:42.7745021Z OK 2022-12-01T10:40:42.7745026Z 2022-12-01T10:40:42.7745109Z Generating XML reports... 2022-12-01T10:40:42.7745469Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101705.xml 2022-12-01T10:40:42.7745744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7745867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7746151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7746296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7746483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5w4_vd5w 2022-12-01T10:40:42.7746680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5w4_vd5w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7746938Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7746943Z 2022-12-01T10:40:42.7747018Z Running tests... 2022-12-01T10:40:42.7747206Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7747447Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23507 2022-12-01T10:40:42.7747607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23508 2022-12-01T10:40:42.7747766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23509 2022-12-01T10:40:42.7747924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23510 2022-12-01T10:40:42.7748202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7748327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7748614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7748764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7748955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0txh2fy7 2022-12-01T10:40:42.7749151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0txh2fy7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7749423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7749575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7749858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7749995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7750181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphk66_at8 2022-12-01T10:40:42.7750377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphk66_at8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7750638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7750762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7751041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7751179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7751368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk_xpozo 2022-12-01T10:40:42.7751564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk_xpozo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7751840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7751964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7752232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7752368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7752553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wg189uo 2022-12-01T10:40:42.7752748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wg189uo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7752916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7753083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7753247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7753410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7753480Z ok (2.136s) 2022-12-01T10:40:42.7753488Z 2022-12-01T10:40:42.7753675Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7753751Z Ran 1 test in 2.136s 2022-12-01T10:40:42.7753756Z 2022-12-01T10:40:42.7753817Z OK 2022-12-01T10:40:42.7753822Z 2022-12-01T10:40:42.7753908Z Generating XML reports... 2022-12-01T10:40:42.7754280Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101709.xml 2022-12-01T10:40:42.7754558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7754683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7754961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7755083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7755293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ryccqst 2022-12-01T10:40:42.7755490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ryccqst/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7755743Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7755748Z 2022-12-01T10:40:42.7755823Z Running tests... 2022-12-01T10:40:42.7756017Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7756287Z test_multiple_backward (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23750 2022-12-01T10:40:42.7756444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23751 2022-12-01T10:40:42.7756600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23752 2022-12-01T10:40:42.7756742Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23753 2022-12-01T10:40:42.7757016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7757139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7757418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7757552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7757740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcz2wmcqf 2022-12-01T10:40:42.7757935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcz2wmcqf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7758328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7758441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7758726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7758863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7759047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvz_s9tev 2022-12-01T10:40:42.7759243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvz_s9tev/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7759514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7759642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7759921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7760058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7760230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuc_7bgym 2022-12-01T10:40:42.7760426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuc_7bgym/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7760700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7760824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7761103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7761241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7761426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd00ird7j 2022-12-01T10:40:42.7761621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd00ird7j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7761785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7761983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7762151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7762311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7762380Z ok (2.839s) 2022-12-01T10:40:42.7762385Z 2022-12-01T10:40:42.7762582Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7762698Z Ran 1 test in 2.840s 2022-12-01T10:40:42.7762704Z 2022-12-01T10:40:42.7762764Z OK 2022-12-01T10:40:42.7762769Z 2022-12-01T10:40:42.7762854Z Generating XML reports... 2022-12-01T10:40:42.7763211Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101712.xml 2022-12-01T10:40:42.7763488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7763618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7763899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7764035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7764221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhnrojwx 2022-12-01T10:40:42.7764422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhnrojwx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7764676Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7764681Z 2022-12-01T10:40:42.7764753Z Running tests... 2022-12-01T10:40:42.7764937Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7765189Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23993 2022-12-01T10:40:42.7765346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23994 2022-12-01T10:40:42.7765499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23995 2022-12-01T10:40:42.7765652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23996 2022-12-01T10:40:42.7765927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7766052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7766332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7766452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7766638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kjoyozu 2022-12-01T10:40:42.7766834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kjoyozu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7767104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7767228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7767509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7767646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7767886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6uiuawu 2022-12-01T10:40:42.7768085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6uiuawu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7768346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7768508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7768790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7768926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7769110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_m_kb1c 2022-12-01T10:40:42.7769303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_m_kb1c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7769603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7769729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7769994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7770132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7770318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypclxb4p 2022-12-01T10:40:42.7770515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypclxb4p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7770681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7770846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7771011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7771173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7771352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7771515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7771691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7771864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7772166Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7772463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7772757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7773045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7773315Z [W tensorpipe_agent.cpp:679] RPC agent for worker2 encountered error when sending response to request #150 to worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7773538Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7773754Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7774008Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.7774268Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7774553Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #150 to worker2, as the agent is shutting down 2022-12-01T10:40:42.7774835Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7775093Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7775377Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #150 to worker3, as the agent is shutting down 2022-12-01T10:40:42.7775686Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #152 to worker0, as the agent is shutting down 2022-12-01T10:40:42.7775937Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7776193Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7776453Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7776671Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7776921Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7777167Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7777418Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7777638Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.7777706Z ok (2.540s) 2022-12-01T10:40:42.7777711Z 2022-12-01T10:40:42.7777914Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7777993Z Ran 1 test in 2.540s 2022-12-01T10:40:42.7777998Z 2022-12-01T10:40:42.7778058Z OK 2022-12-01T10:40:42.7778062Z 2022-12-01T10:40:42.7778148Z Generating XML reports... 2022-12-01T10:40:42.7778520Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101717.xml 2022-12-01T10:40:42.7778796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7778911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7779192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7779327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7779516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryif4qwg 2022-12-01T10:40:42.7779712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryif4qwg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7779967Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7779972Z 2022-12-01T10:40:42.7780046Z Running tests... 2022-12-01T10:40:42.7780240Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7780537Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24248 2022-12-01T10:40:42.7780696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24249 2022-12-01T10:40:42.7780850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24250 2022-12-01T10:40:42.7781002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24251 2022-12-01T10:40:42.7781278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7781427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7781708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7781842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7782027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtt7wuqw 2022-12-01T10:40:42.7782213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtt7wuqw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7782484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7782606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7782887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7783025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7783213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpef4r2zmw 2022-12-01T10:40:42.7783411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpef4r2zmw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7783681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7783807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7784073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7784208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7784393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50yy25rr 2022-12-01T10:40:42.7784586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50yy25rr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7784858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7784980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7785257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7785390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7785561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllakiweg 2022-12-01T10:40:42.7785755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllakiweg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7785919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7786085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7786249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7786412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7786479Z ok (2.037s) 2022-12-01T10:40:42.7786484Z 2022-12-01T10:40:42.7786684Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7786747Z Ran 1 test in 2.038s 2022-12-01T10:40:42.7786764Z 2022-12-01T10:40:42.7786811Z OK 2022-12-01T10:40:42.7786815Z 2022-12-01T10:40:42.7786942Z Generating XML reports... 2022-12-01T10:40:42.7787311Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101720.xml 2022-12-01T10:40:42.7787584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7787707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7788015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7788149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7788332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n36x_vh 2022-12-01T10:40:42.7788511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n36x_vh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7788770Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7788775Z 2022-12-01T10:40:42.7788847Z Running tests... 2022-12-01T10:40:42.7789042Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7789277Z test_nested_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24491 2022-12-01T10:40:42.7789434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24492 2022-12-01T10:40:42.7789593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24493 2022-12-01T10:40:42.7789747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24494 2022-12-01T10:40:42.7790021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7790132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7790413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7790548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7790732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cahf7w7 2022-12-01T10:40:42.7790930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cahf7w7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7791207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7791330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7791611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7791732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7791921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrta8ho_ 2022-12-01T10:40:42.7792118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrta8ho_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7792388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7792511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7792791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7792927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7793113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1prqh5s5 2022-12-01T10:40:42.7793306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1prqh5s5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7793595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7793717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7793996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7794129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7794314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyng5fvbr 2022-12-01T10:40:42.7794535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyng5fvbr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7794700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7794863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7795014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7795176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7795245Z ok (2.037s) 2022-12-01T10:40:42.7795250Z 2022-12-01T10:40:42.7795447Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7795524Z Ran 1 test in 2.038s 2022-12-01T10:40:42.7795529Z 2022-12-01T10:40:42.7795589Z OK 2022-12-01T10:40:42.7795593Z 2022-12-01T10:40:42.7795676Z Generating XML reports... 2022-12-01T10:40:42.7796048Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101724.xml 2022-12-01T10:40:42.7796321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7796433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7796714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7796852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7797037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvawexd9x 2022-12-01T10:40:42.7797232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvawexd9x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7797486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7797493Z 2022-12-01T10:40:42.7797567Z Running tests... 2022-12-01T10:40:42.7797763Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7797887Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest) 2022-12-01T10:40:42.7798087Z Similar to test in test_autograd.py. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24714 2022-12-01T10:40:42.7798345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24715 2022-12-01T10:40:42.7798505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24716 2022-12-01T10:40:42.7798660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24717 2022-12-01T10:40:42.7798937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7799061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7799344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7799482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7799654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a_ll6l6 2022-12-01T10:40:42.7799849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a_ll6l6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7800172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7800297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7800575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7800711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7800896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepn_d8wy 2022-12-01T10:40:42.7801129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepn_d8wy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7801387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7801511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7801791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7801929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7802113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpit6aluwy 2022-12-01T10:40:42.7802310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpit6aluwy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7802583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7802711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7802992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7803114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7803301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfh3jdwa0 2022-12-01T10:40:42.7803498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfh3jdwa0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7803666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7803833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7803998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7804161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7804234Z ok (2.041s) 2022-12-01T10:40:42.7804239Z 2022-12-01T10:40:42.7804426Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7804506Z Ran 1 test in 2.041s 2022-12-01T10:40:42.7804510Z 2022-12-01T10:40:42.7804573Z OK 2022-12-01T10:40:42.7804577Z 2022-12-01T10:40:42.7804664Z Generating XML reports... 2022-12-01T10:40:42.7805040Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101727.xml 2022-12-01T10:40:42.7805317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7805443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7805725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7805864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7806038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzap4glv 2022-12-01T10:40:42.7806235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzap4glv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7806493Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7806498Z 2022-12-01T10:40:42.7806573Z Running tests... 2022-12-01T10:40:42.7806795Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7807039Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24957 2022-12-01T10:40:42.7807196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24958 2022-12-01T10:40:42.7807353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24959 2022-12-01T10:40:42.7807539Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24960 2022-12-01T10:40:42.7807855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7807984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7808268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7808408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7808594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1uck_ji 2022-12-01T10:40:42.7808789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1uck_ji/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7809065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7809193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7809463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7809599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7809784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx23740hm 2022-12-01T10:40:42.7809979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx23740hm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7810253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7810377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7810657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7810792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7810979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qs9v6le 2022-12-01T10:40:42.7811160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qs9v6le/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7811432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7811555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7811835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7811971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7812152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v8hjch9 2022-12-01T10:40:42.7812346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v8hjch9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7812512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7812663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7812828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7812989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7813516Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-12-01T10:40:42.7813655Z "Argument order of nn.functional.embedding_bag was changed. " 2022-12-01T10:40:42.7814144Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-12-01T10:40:42.7814306Z "Argument order of nn.functional.embedding_bag was changed. " 2022-12-01T10:40:42.7814792Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-12-01T10:40:42.7814928Z "Argument order of nn.functional.embedding_bag was changed. " 2022-12-01T10:40:42.7815412Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-12-01T10:40:42.7815545Z "Argument order of nn.functional.embedding_bag was changed. " 2022-12-01T10:40:42.7815599Z ok (2.141s) 2022-12-01T10:40:42.7815604Z 2022-12-01T10:40:42.7815805Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7815882Z Ran 1 test in 2.142s 2022-12-01T10:40:42.7815887Z 2022-12-01T10:40:42.7815948Z OK 2022-12-01T10:40:42.7815953Z 2022-12-01T10:40:42.7816038Z Generating XML reports... 2022-12-01T10:40:42.7816406Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101731.xml 2022-12-01T10:40:42.7816685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7816810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7817077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7817214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7817401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpny7rd3 2022-12-01T10:40:42.7817602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpny7rd3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7817857Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7817862Z 2022-12-01T10:40:42.7817934Z Running tests... 2022-12-01T10:40:42.7818130Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7818387Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25212 2022-12-01T10:40:42.7818551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25213 2022-12-01T10:40:42.7818693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25214 2022-12-01T10:40:42.7818847Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25215 2022-12-01T10:40:42.7819121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7819246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7819525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7819666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7819883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuotb2i6g 2022-12-01T10:40:42.7820083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuotb2i6g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7820356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7820467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7820747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7820916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7821102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicfcdspj 2022-12-01T10:40:42.7821298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicfcdspj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7821571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7821694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7821973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7822095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7822278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7afdr7h 2022-12-01T10:40:42.7822475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7afdr7h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7822747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7822869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7823145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7823281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7823463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb17v5r2 2022-12-01T10:40:42.7823656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb17v5r2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7823807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7823973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7824138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7824299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7824481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7824662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7824837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7825012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7825313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7825603Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7825893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7826183Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7826251Z ok (2.237s) 2022-12-01T10:40:42.7826257Z 2022-12-01T10:40:42.7826484Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7826561Z Ran 1 test in 2.237s 2022-12-01T10:40:42.7826566Z 2022-12-01T10:40:42.7826625Z OK 2022-12-01T10:40:42.7826630Z 2022-12-01T10:40:42.7826714Z Generating XML reports... 2022-12-01T10:40:42.7827070Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101735.xml 2022-12-01T10:40:42.7827345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7827497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7827779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7827915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7828101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mmkqd9a 2022-12-01T10:40:42.7828299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mmkqd9a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7828554Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7828559Z 2022-12-01T10:40:42.7828631Z Running tests... 2022-12-01T10:40:42.7828816Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7829082Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25447 2022-12-01T10:40:42.7829239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25448 2022-12-01T10:40:42.7829393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25449 2022-12-01T10:40:42.7829546Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25450 2022-12-01T10:40:42.7829825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7829950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7830230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7830364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7830538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nnkkvpf 2022-12-01T10:40:42.7830737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nnkkvpf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7831008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7831130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7831413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7831547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7831735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4wztktk 2022-12-01T10:40:42.7831930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4wztktk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7832188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7832312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7832592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7832725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7832909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpop6j32aa 2022-12-01T10:40:42.7833150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpop6j32aa/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7833422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7833545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7833821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7833971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7834156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf11smhu_ 2022-12-01T10:40:42.7834352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf11smhu_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7834516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7834681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7834844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7854626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7854886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.7855074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.7855260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.7855421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.7855780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7856089Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7856384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7856678Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.7856748Z ok (2.339s) 2022-12-01T10:40:42.7856757Z 2022-12-01T10:40:42.7856957Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7857035Z Ran 1 test in 2.339s 2022-12-01T10:40:42.7857041Z 2022-12-01T10:40:42.7857102Z OK 2022-12-01T10:40:42.7857107Z 2022-12-01T10:40:42.7857179Z Generating XML reports... 2022-12-01T10:40:42.7857558Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101738.xml 2022-12-01T10:40:42.7857839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7857967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7858251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7858390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7858577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dq4mu8w 2022-12-01T10:40:42.7858780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dq4mu8w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7859021Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7859039Z 2022-12-01T10:40:42.7859100Z Running tests... 2022-12-01T10:40:42.7859297Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7859677Z test_post_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25682 2022-12-01T10:40:42.7859838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25683 2022-12-01T10:40:42.7859995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25684 2022-12-01T10:40:42.7860150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25685 2022-12-01T10:40:42.7860476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7860602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7860872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7861009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7861200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvk2n9vyf 2022-12-01T10:40:42.7861398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvk2n9vyf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7861671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7861794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7862074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7862213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7862384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_ggh0mf 2022-12-01T10:40:42.7862581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_ggh0mf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7862855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7862982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7863263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7863398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7863583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9zia081 2022-12-01T10:40:42.7863780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9zia081/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7864049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7864158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7864439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7864577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7864760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hzfpkst 2022-12-01T10:40:42.7864956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hzfpkst/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7865119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7865288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7865449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7865609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7865664Z ok (2.039s) 2022-12-01T10:40:42.7865669Z 2022-12-01T10:40:42.7865866Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7865941Z Ran 1 test in 2.039s 2022-12-01T10:40:42.7865984Z 2022-12-01T10:40:42.7866048Z OK 2022-12-01T10:40:42.7866052Z 2022-12-01T10:40:42.7866136Z Generating XML reports... 2022-12-01T10:40:42.7866507Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101742.xml 2022-12-01T10:40:42.7866780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7866930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7867199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7867333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7867517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqwx7g0w 2022-12-01T10:40:42.7867715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqwx7g0w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7867972Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7867977Z 2022-12-01T10:40:42.7868049Z Running tests... 2022-12-01T10:40:42.7868247Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7868490Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25925 2022-12-01T10:40:42.7868652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25926 2022-12-01T10:40:42.7868795Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25927 2022-12-01T10:40:42.7868949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25928 2022-12-01T10:40:42.7869223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7869349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7869632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7869768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7869954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93kvazxo 2022-12-01T10:40:42.7870151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93kvazxo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7870410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7870533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7870810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7870947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7871131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoksrlm0r 2022-12-01T10:40:42.7871325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoksrlm0r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7871596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7871717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7871997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7872119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7872301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rpcw02o 2022-12-01T10:40:42.7872495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rpcw02o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7872796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7872919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7873198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7873332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7873543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqh53jsb 2022-12-01T10:40:42.7873725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqh53jsb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7873891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7874053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7874217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7874380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7874446Z ok (2.137s) 2022-12-01T10:40:42.7874452Z 2022-12-01T10:40:42.7874652Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7874727Z Ran 1 test in 2.137s 2022-12-01T10:40:42.7874732Z 2022-12-01T10:40:42.7874795Z OK 2022-12-01T10:40:42.7874801Z 2022-12-01T10:40:42.7874872Z Generating XML reports... 2022-12-01T10:40:42.7875240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101745.xml 2022-12-01T10:40:42.7875511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7875633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7875917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7876051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7876235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusmsexc6 2022-12-01T10:40:42.7876461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusmsexc6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7876801Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7876824Z 2022-12-01T10:40:42.7876886Z Running tests... 2022-12-01T10:40:42.7877083Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7877323Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26148 2022-12-01T10:40:42.7877479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26149 2022-12-01T10:40:42.7877637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26150 2022-12-01T10:40:42.7877791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26151 2022-12-01T10:40:42.7878067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7878300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7878576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7878710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7878900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ebahrrp 2022-12-01T10:40:42.7879173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7879354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7879640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7879777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7879965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs2n179m 2022-12-01T10:40:42.7880163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ebahrrp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7880375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs2n179m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7880648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7880773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7881057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7881193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7881379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4ivk5pb 2022-12-01T10:40:42.7881573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4ivk5pb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7881843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7881954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7882232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7882367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7882551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphoruh_sj 2022-12-01T10:40:42.7882744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphoruh_sj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7882910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7883076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7883238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7883403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7883458Z ok (2.139s) 2022-12-01T10:40:42.7883465Z 2022-12-01T10:40:42.7883662Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7883736Z Ran 1 test in 2.139s 2022-12-01T10:40:42.7883741Z 2022-12-01T10:40:42.7883800Z OK 2022-12-01T10:40:42.7883805Z 2022-12-01T10:40:42.7883887Z Generating XML reports... 2022-12-01T10:40:42.7884260Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101749.xml 2022-12-01T10:40:42.7884536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7884660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7884928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7885067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7885253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopqpaujr 2022-12-01T10:40:42.7885452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopqpaujr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7885707Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7885713Z 2022-12-01T10:40:42.7885813Z Running tests... 2022-12-01T10:40:42.7886011Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7886256Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26371 2022-12-01T10:40:42.7886412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26372 2022-12-01T10:40:42.7886556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26373 2022-12-01T10:40:42.7886737Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26374 2022-12-01T10:40:42.7887017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7887139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7887422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7887561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7887800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf_cxsuq 2022-12-01T10:40:42.7888002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf_cxsuq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7888265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7888393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7888675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7888811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7888996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfi_2m_r 2022-12-01T10:40:42.7889193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfi_2m_r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7889465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7889590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7889871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7889997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7890184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulcqh53g 2022-12-01T10:40:42.7890383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulcqh53g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7890655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7890778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7891060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7891194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7891377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i061929 2022-12-01T10:40:42.7891568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i061929/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7891722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7891884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7892047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7892205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7892272Z ok (3.139s) 2022-12-01T10:40:42.7892312Z 2022-12-01T10:40:42.7892510Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7892587Z Ran 1 test in 3.139s 2022-12-01T10:40:42.7892592Z 2022-12-01T10:40:42.7892655Z OK 2022-12-01T10:40:42.7892660Z 2022-12-01T10:40:42.7892731Z Generating XML reports... 2022-12-01T10:40:42.7893102Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101753.xml 2022-12-01T10:40:42.7893402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7893527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7893807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7893942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7894130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx693hwei 2022-12-01T10:40:42.7894325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx693hwei/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7894579Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7894585Z 2022-12-01T10:40:42.7894645Z Running tests... 2022-12-01T10:40:42.7894841Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7895073Z test_trainer_ps (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26614 2022-12-01T10:40:42.7895230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26615 2022-12-01T10:40:42.7895385Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26616 2022-12-01T10:40:42.7895540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26617 2022-12-01T10:40:42.7895818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7895940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7896209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7896343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7896531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2302a0j 2022-12-01T10:40:42.7896723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2302a0j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7896994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7897117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7897400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7897534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7897719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcoxwjq96 2022-12-01T10:40:42.7897901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcoxwjq96/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7898172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7898295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7898573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7898708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7898933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwwm4com 2022-12-01T10:40:42.7899129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwwm4com/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7899397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7899507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7899788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7899954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7900140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1ttvw5f 2022-12-01T10:40:42.7900334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1ttvw5f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7900498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7900663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7900824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7900984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7901039Z ok (2.340s) 2022-12-01T10:40:42.7901044Z 2022-12-01T10:40:42.7901244Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7901321Z Ran 1 test in 2.340s 2022-12-01T10:40:42.7901326Z 2022-12-01T10:40:42.7901386Z OK 2022-12-01T10:40:42.7901390Z 2022-12-01T10:40:42.7901473Z Generating XML reports... 2022-12-01T10:40:42.7901843Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101757.xml 2022-12-01T10:40:42.7902118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7902243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7902509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7902645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7902829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfbbhiwr 2022-12-01T10:40:42.7903027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfbbhiwr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7903283Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7903289Z 2022-12-01T10:40:42.7903362Z Running tests... 2022-12-01T10:40:42.7903559Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7903821Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26857 2022-12-01T10:40:42.7903978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26858 2022-12-01T10:40:42.7904120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26859 2022-12-01T10:40:42.7904272Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26860 2022-12-01T10:40:42.7904547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7904672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7904954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7905090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7905275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp895091ah 2022-12-01T10:40:42.7905497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp895091ah/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7905769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7905881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7906159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7906321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7906508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphdvpc0fj 2022-12-01T10:40:42.7906703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphdvpc0fj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7906972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7907097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7907378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7907499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7907681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfncyqtxv 2022-12-01T10:40:42.7907879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfncyqtxv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7908148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7908271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7908548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7908683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7908867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf74f5nz 2022-12-01T10:40:42.7909062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf74f5nz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7909213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7909376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7909541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7909701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7909770Z ok (2.345s) 2022-12-01T10:40:42.7909775Z 2022-12-01T10:40:42.7909975Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7910052Z Ran 1 test in 2.346s 2022-12-01T10:40:42.7910057Z 2022-12-01T10:40:42.7910119Z OK 2022-12-01T10:40:42.7910124Z 2022-12-01T10:40:42.7910195Z Generating XML reports... 2022-12-01T10:40:42.7910564Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101801.xml 2022-12-01T10:40:42.7910836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7910960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7911242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7911376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7911563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcbz9o0o 2022-12-01T10:40:42.7911760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcbz9o0o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7912043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7912049Z 2022-12-01T10:40:42.7912110Z Running tests... 2022-12-01T10:40:42.7912308Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7912549Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27100 2022-12-01T10:40:42.7912735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27101 2022-12-01T10:40:42.7912893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27102 2022-12-01T10:40:42.7913047Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27103 2022-12-01T10:40:42.7913321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7913446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7913716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7913851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7914038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjeiih913 2022-12-01T10:40:42.7914232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjeiih913/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7914505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7914628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7914906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7915041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7915227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztw2_qal 2022-12-01T10:40:42.7915411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztw2_qal/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7915682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7915804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7916086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7916222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7916407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2tu1om8 2022-12-01T10:40:42.7916601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2tu1om8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7916872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7916983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7917257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7917391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7917576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfkumm4m 2022-12-01T10:40:42.7917771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfkumm4m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7917936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7918106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7918484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7918652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7918707Z ok (2.138s) 2022-12-01T10:40:42.7918712Z 2022-12-01T10:40:42.7918915Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7918991Z Ran 1 test in 2.138s 2022-12-01T10:40:42.7918996Z 2022-12-01T10:40:42.7919058Z OK 2022-12-01T10:40:42.7919062Z 2022-12-01T10:40:42.7919185Z Generating XML reports... 2022-12-01T10:40:42.7919556Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101805.xml 2022-12-01T10:40:42.7919831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7919957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7920223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7920360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7920546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvurzoekr 2022-12-01T10:40:42.7920745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvurzoekr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7920999Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7921007Z 2022-12-01T10:40:42.7921082Z Running tests... 2022-12-01T10:40:42.7921277Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7921966Z test_dist_optim (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72997 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.219s) 2022-12-01T10:40:42.7921972Z 2022-12-01T10:40:42.7922163Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7922237Z Ran 1 test in 0.220s 2022-12-01T10:40:42.7922242Z 2022-12-01T10:40:42.7922300Z OK (skipped=1) 2022-12-01T10:40:42.7922318Z 2022-12-01T10:40:42.7922390Z Generating XML reports... 2022-12-01T10:40:42.7922765Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221201101808.xml 2022-12-01T10:40:42.7923042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7923166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7923449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7923587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7923773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz2utluj 2022-12-01T10:40:42.7923968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz2utluj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7924211Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7924229Z 2022-12-01T10:40:42.7924291Z Running tests... 2022-12-01T10:40:42.7924486Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7924729Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27333 2022-12-01T10:40:42.7924887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27334 2022-12-01T10:40:42.7925041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27335 2022-12-01T10:40:42.7925221Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27336 2022-12-01T10:40:42.7925500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7925623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7925890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7926087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7926273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kbrgq9t 2022-12-01T10:40:42.7926468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kbrgq9t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7926740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7926865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7927142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7927277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7927448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuug9m1c 2022-12-01T10:40:42.7927644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuug9m1c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7927982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7928108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7928388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7928523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7928711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjqkieas 2022-12-01T10:40:42.7928907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjqkieas/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7929180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7929291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7929571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7929706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7929891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimb4pg9t 2022-12-01T10:40:42.7930086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimb4pg9t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7930252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7930417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7930580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7930729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7931523Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-12-01T10:40:42.7931622Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.7931761Z ValueError('Error running optimizer.') 2022-12-01T10:40:42.7931889Z Traceback (most recent call last): 2022-12-01T10:40:42.7932155Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7932291Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7932571Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-12-01T10:40:42.7932707Z local_optim.step(autograd_ctx_id) 2022-12-01T10:40:42.7932947Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-12-01T10:40:42.7933030Z self.optim.step() 2022-12-01T10:40:42.7933268Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper 2022-12-01T10:40:42.7933350Z out = func(*args, **kwargs) 2022-12-01T10:40:42.7933644Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-12-01T10:40:42.7933758Z raise ValueError("Error running optimizer.") 2022-12-01T10:40:42.7933855Z ValueError: Error running optimizer. 2022-12-01T10:40:42.7933860Z 2022-12-01T10:40:42.7934642Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-12-01T10:40:42.7934738Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.7934873Z ValueError('Error running optimizer.') 2022-12-01T10:40:42.7934952Z Traceback (most recent call last): 2022-12-01T10:40:42.7935213Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7935350Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7935627Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-12-01T10:40:42.7935722Z local_optim.step(autograd_ctx_id) 2022-12-01T10:40:42.7935974Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-12-01T10:40:42.7936055Z self.optim.step() 2022-12-01T10:40:42.7936279Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper 2022-12-01T10:40:42.7936361Z out = func(*args, **kwargs) 2022-12-01T10:40:42.7936650Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-12-01T10:40:42.7936764Z raise ValueError("Error running optimizer.") 2022-12-01T10:40:42.7936862Z ValueError: Error running optimizer. 2022-12-01T10:40:42.7936867Z 2022-12-01T10:40:42.7937665Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-12-01T10:40:42.7938440Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-12-01T10:40:42.7938534Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.7938671Z ValueError('Error running optimizer.') 2022-12-01T10:40:42.7938762Z Traceback (most recent call last): 2022-12-01T10:40:42.7939052Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7939174Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7939454Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-12-01T10:40:42.7939549Z local_optim.step(autograd_ctx_id) 2022-12-01T10:40:42.7939830Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-12-01T10:40:42.7939910Z self.optim.step() 2022-12-01T10:40:42.7940144Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper 2022-12-01T10:40:42.7940226Z out = func(*args, **kwargs) 2022-12-01T10:40:42.7940516Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-12-01T10:40:42.7940619Z raise ValueError("Error running optimizer.") 2022-12-01T10:40:42.7940716Z ValueError: Error running optimizer. 2022-12-01T10:40:42.7940721Z 2022-12-01T10:40:42.7940812Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.7940948Z ValueError('Error running optimizer.') 2022-12-01T10:40:42.7941039Z Traceback (most recent call last): 2022-12-01T10:40:42.7941297Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7941432Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7941712Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-12-01T10:40:42.7941795Z local_optim.step(autograd_ctx_id) 2022-12-01T10:40:42.7942043Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-12-01T10:40:42.7942126Z self.optim.step() 2022-12-01T10:40:42.7942362Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper 2022-12-01T10:40:42.7942444Z out = func(*args, **kwargs) 2022-12-01T10:40:42.7942736Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-12-01T10:40:42.7942848Z raise ValueError("Error running optimizer.") 2022-12-01T10:40:42.7942932Z ValueError: Error running optimizer. 2022-12-01T10:40:42.7942949Z 2022-12-01T10:40:42.7943029Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.7943164Z ValueError('Error running optimizer.') 2022-12-01T10:40:42.7943253Z Traceback (most recent call last): 2022-12-01T10:40:42.7943511Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7943644Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7943924Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-12-01T10:40:42.7944019Z local_optim.step(autograd_ctx_id) 2022-12-01T10:40:42.7944258Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-12-01T10:40:42.7944336Z self.optim.step() 2022-12-01T10:40:42.7944571Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper 2022-12-01T10:40:42.7944653Z out = func(*args, **kwargs) 2022-12-01T10:40:42.7944946Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-12-01T10:40:42.7945059Z raise ValueError("Error running optimizer.") 2022-12-01T10:40:42.7945157Z ValueError: Error running optimizer. 2022-12-01T10:40:42.7945162Z 2022-12-01T10:40:42.7945250Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.7945373Z ValueError('Error running optimizer.') 2022-12-01T10:40:42.7945462Z Traceback (most recent call last): 2022-12-01T10:40:42.7945762Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7945898Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7946176Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-12-01T10:40:42.7946271Z local_optim.step(autograd_ctx_id) 2022-12-01T10:40:42.7946521Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-12-01T10:40:42.7946631Z self.optim.step() 2022-12-01T10:40:42.7946854Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper 2022-12-01T10:40:42.7946936Z out = func(*args, **kwargs) 2022-12-01T10:40:42.7947224Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-12-01T10:40:42.7947336Z raise ValueError("Error running optimizer.") 2022-12-01T10:40:42.7947435Z ValueError: Error running optimizer. 2022-12-01T10:40:42.7947441Z 2022-12-01T10:40:42.7947533Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.7947670Z ValueError('Error running optimizer.') 2022-12-01T10:40:42.7947762Z Traceback (most recent call last): 2022-12-01T10:40:42.7948008Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7948143Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7948424Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-12-01T10:40:42.7948521Z local_optim.step(autograd_ctx_id) 2022-12-01T10:40:42.7948777Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-12-01T10:40:42.7948858Z self.optim.step() 2022-12-01T10:40:42.7949095Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper 2022-12-01T10:40:42.7949176Z out = func(*args, **kwargs) 2022-12-01T10:40:42.7949451Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-12-01T10:40:42.7949563Z raise ValueError("Error running optimizer.") 2022-12-01T10:40:42.7949659Z ValueError: Error running optimizer. 2022-12-01T10:40:42.7949665Z 2022-12-01T10:40:42.7949753Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.7949892Z ValueError('Error running optimizer.') 2022-12-01T10:40:42.7949982Z Traceback (most recent call last): 2022-12-01T10:40:42.7950241Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7950361Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7950638Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-12-01T10:40:42.7950734Z local_optim.step(autograd_ctx_id) 2022-12-01T10:40:42.7950987Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-12-01T10:40:42.7951068Z self.optim.step() 2022-12-01T10:40:42.7951301Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper 2022-12-01T10:40:42.7951383Z out = func(*args, **kwargs) 2022-12-01T10:40:42.7951673Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-12-01T10:40:42.7951775Z raise ValueError("Error running optimizer.") 2022-12-01T10:40:42.7951873Z ValueError: Error running optimizer. 2022-12-01T10:40:42.7951878Z 2022-12-01T10:40:42.7951945Z ok (2.140s) 2022-12-01T10:40:42.7951950Z 2022-12-01T10:40:42.7952150Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7952225Z Ran 1 test in 2.141s 2022-12-01T10:40:42.7952230Z 2022-12-01T10:40:42.7952290Z OK 2022-12-01T10:40:42.7952323Z 2022-12-01T10:40:42.7952411Z Generating XML reports... 2022-12-01T10:40:42.7952791Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221201101810.xml 2022-12-01T10:40:42.7953067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7953179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7953493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7953630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7953818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy8171oi 2022-12-01T10:40:42.7954016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy8171oi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7954274Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7954279Z 2022-12-01T10:40:42.7954351Z Running tests... 2022-12-01T10:40:42.7954551Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7954811Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27576 2022-12-01T10:40:42.7954959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27577 2022-12-01T10:40:42.7955113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27578 2022-12-01T10:40:42.7955267Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27579 2022-12-01T10:40:42.7955541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7955666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7955948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7956085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7956270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dqco2gw 2022-12-01T10:40:42.7956453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dqco2gw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7956725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7956848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7957126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7957258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7957442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c_yy9bx 2022-12-01T10:40:42.7957636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c_yy9bx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7957907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7958030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7958428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7958565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7958748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vwo5sns 2022-12-01T10:40:42.7958943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vwo5sns/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7959260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7959385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7959667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7959801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7959975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3ovx0eq 2022-12-01T10:40:42.7960207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3ovx0eq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7960374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7960542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7960705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7960871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7961697Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-12-01T10:40:42.7961793Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.7961933Z ValueError('Error creating optimizer.') 2022-12-01T10:40:42.7962025Z Traceback (most recent call last): 2022-12-01T10:40:42.7962276Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7962411Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7962693Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-12-01T10:40:42.7962828Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-12-01T10:40:42.7963087Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-12-01T10:40:42.7963154Z **kwargs) 2022-12-01T10:40:42.7963449Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-12-01T10:40:42.7963569Z raise ValueError("Error creating optimizer.") 2022-12-01T10:40:42.7963656Z ValueError: Error creating optimizer. 2022-12-01T10:40:42.7963661Z 2022-12-01T10:40:42.7964485Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-12-01T10:40:42.7965293Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-12-01T10:40:42.7965390Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.7965530Z ValueError('Error creating optimizer.') 2022-12-01T10:40:42.7965624Z Traceback (most recent call last): 2022-12-01T10:40:42.7965884Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7966049Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7966330Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-12-01T10:40:42.7966465Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-12-01T10:40:42.7966720Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-12-01T10:40:42.7966804Z **kwargs) 2022-12-01T10:40:42.7967098Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-12-01T10:40:42.7967210Z raise ValueError("Error creating optimizer.") 2022-12-01T10:40:42.7967310Z ValueError: Error creating optimizer. 2022-12-01T10:40:42.7967314Z 2022-12-01T10:40:42.7968208Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-12-01T10:40:42.7968304Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.7968449Z ValueError('Error creating optimizer.') 2022-12-01T10:40:42.7968542Z Traceback (most recent call last): 2022-12-01T10:40:42.7968805Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7968926Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7969204Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-12-01T10:40:42.7969336Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-12-01T10:40:42.7969598Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-12-01T10:40:42.7969666Z **kwargs) 2022-12-01T10:40:42.7969963Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-12-01T10:40:42.7970077Z raise ValueError("Error creating optimizer.") 2022-12-01T10:40:42.7970177Z ValueError: Error creating optimizer. 2022-12-01T10:40:42.7970182Z 2022-12-01T10:40:42.7970260Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.7970400Z ValueError('Error creating optimizer.') 2022-12-01T10:40:42.7970491Z Traceback (most recent call last): 2022-12-01T10:40:42.7970749Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7970881Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7971156Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-12-01T10:40:42.7971290Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-12-01T10:40:42.7971546Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-12-01T10:40:42.7971600Z **kwargs) 2022-12-01T10:40:42.7971896Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-12-01T10:40:42.7972011Z raise ValueError("Error creating optimizer.") 2022-12-01T10:40:42.7972110Z ValueError: Error creating optimizer. 2022-12-01T10:40:42.7972115Z 2022-12-01T10:40:42.7972205Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.7972344Z ValueError('Error creating optimizer.') 2022-12-01T10:40:42.7972438Z Traceback (most recent call last): 2022-12-01T10:40:42.7972696Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7972850Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7973127Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-12-01T10:40:42.7973259Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-12-01T10:40:42.7973514Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-12-01T10:40:42.7973580Z **kwargs) 2022-12-01T10:40:42.7973873Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-12-01T10:40:42.7974015Z raise ValueError("Error creating optimizer.") 2022-12-01T10:40:42.7974113Z ValueError: Error creating optimizer. 2022-12-01T10:40:42.7974117Z 2022-12-01T10:40:42.7974195Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.7974333Z ValueError('Error creating optimizer.') 2022-12-01T10:40:42.7974424Z Traceback (most recent call last): 2022-12-01T10:40:42.7974687Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7974821Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7975097Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-12-01T10:40:42.7975228Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-12-01T10:40:42.7975482Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-12-01T10:40:42.7975538Z **kwargs) 2022-12-01T10:40:42.7975831Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-12-01T10:40:42.7975943Z raise ValueError("Error creating optimizer.") 2022-12-01T10:40:42.7976040Z ValueError: Error creating optimizer. 2022-12-01T10:40:42.7976046Z 2022-12-01T10:40:42.7976135Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.7976276Z ValueError('Error creating optimizer.') 2022-12-01T10:40:42.7976366Z Traceback (most recent call last): 2022-12-01T10:40:42.7976625Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7976743Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7977016Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-12-01T10:40:42.7977150Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-12-01T10:40:42.7977405Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-12-01T10:40:42.7977470Z **kwargs) 2022-12-01T10:40:42.7977768Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-12-01T10:40:42.7977878Z raise ValueError("Error creating optimizer.") 2022-12-01T10:40:42.7977978Z ValueError: Error creating optimizer. 2022-12-01T10:40:42.7977983Z 2022-12-01T10:40:42.7978060Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.7978198Z ValueError('Error creating optimizer.') 2022-12-01T10:40:42.7978288Z Traceback (most recent call last): 2022-12-01T10:40:42.7978547Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.7978679Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.7978956Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-12-01T10:40:42.7979089Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-12-01T10:40:42.7979344Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-12-01T10:40:42.7979396Z **kwargs) 2022-12-01T10:40:42.7979714Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-12-01T10:40:42.7979828Z raise ValueError("Error creating optimizer.") 2022-12-01T10:40:42.7979925Z ValueError: Error creating optimizer. 2022-12-01T10:40:42.7979930Z 2022-12-01T10:40:42.7979996Z ok (2.138s) 2022-12-01T10:40:42.7980002Z 2022-12-01T10:40:42.7980202Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7980277Z Ran 1 test in 2.139s 2022-12-01T10:40:42.7980313Z 2022-12-01T10:40:42.7980374Z OK 2022-12-01T10:40:42.7980379Z 2022-12-01T10:40:42.7980451Z Generating XML reports... 2022-12-01T10:40:42.7980825Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221201101813.xml 2022-12-01T10:40:42.7981098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7981225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7981506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7981642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7981829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdg9ath3d 2022-12-01T10:40:42.7982027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdg9ath3d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7982285Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7982290Z 2022-12-01T10:40:42.7982350Z Running tests... 2022-12-01T10:40:42.7982547Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7983249Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77075 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.219s) 2022-12-01T10:40:42.7983255Z 2022-12-01T10:40:42.7983451Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7983526Z Ran 1 test in 0.219s 2022-12-01T10:40:42.7983531Z 2022-12-01T10:40:42.7983602Z OK (skipped=1) 2022-12-01T10:40:42.7983609Z 2022-12-01T10:40:42.7983697Z Generating XML reports... 2022-12-01T10:40:42.7984068Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221201101817.xml 2022-12-01T10:40:42.7984344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7984469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7984743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7984879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7985066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6m58ids9 2022-12-01T10:40:42.7985263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6m58ids9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7985518Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7985525Z 2022-12-01T10:40:42.7985598Z Running tests... 2022-12-01T10:40:42.7985795Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7986036Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27809 2022-12-01T10:40:42.7986193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27810 2022-12-01T10:40:42.7986366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27811 2022-12-01T10:40:42.7986524Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27812 2022-12-01T10:40:42.7986798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7986922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7987233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7987369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7987557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal85a9wf 2022-12-01T10:40:42.7987753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal85a9wf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7988013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7988135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7988421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7988556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7988740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_82vw__d 2022-12-01T10:40:42.7988935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_82vw__d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7989209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7989330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7989613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7989735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7989918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cv17qdk 2022-12-01T10:40:42.7990111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cv17qdk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7990382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7990507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7990786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7990920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7991103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0j2r1jx6 2022-12-01T10:40:42.7991284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0j2r1jx6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7991450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.7991614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.7991776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.7991939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.7992005Z ok (2.138s) 2022-12-01T10:40:42.7992011Z 2022-12-01T10:40:42.7992208Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7992283Z Ran 1 test in 2.139s 2022-12-01T10:40:42.7992288Z 2022-12-01T10:40:42.7992335Z OK 2022-12-01T10:40:42.7992352Z 2022-12-01T10:40:42.7992422Z Generating XML reports... 2022-12-01T10:40:42.7992849Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221201101819.xml 2022-12-01T10:40:42.7993126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7993250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7993531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7993691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7993876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgdda3ec 2022-12-01T10:40:42.7994073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgdda3ec/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7994315Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.7994333Z 2022-12-01T10:40:42.7994396Z Running tests... 2022-12-01T10:40:42.7994594Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.7994838Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28042 2022-12-01T10:40:42.7994994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28043 2022-12-01T10:40:42.7995150Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28044 2022-12-01T10:40:42.7995304Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28045 2022-12-01T10:40:42.7995580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7995706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7995976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7996114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7996301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfyw6_xa 2022-12-01T10:40:42.7996574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7996697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7996979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7997116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7997300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmih35o6e 2022-12-01T10:40:42.7997482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfyw6_xa/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7997671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmih35o6e/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7997943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7998066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7998454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7998590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.7998777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwo6v_vhg 2022-12-01T10:40:42.7998971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwo6v_vhg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.7999244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.7999354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.7999684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.7999818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8000001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc85q9dk 2022-12-01T10:40:42.8000193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc85q9dk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8000395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8000561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8000724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8000874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8000942Z ok (2.138s) 2022-12-01T10:40:42.8000950Z 2022-12-01T10:40:42.8001148Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8001224Z Ran 1 test in 2.138s 2022-12-01T10:40:42.8001229Z 2022-12-01T10:40:42.8001291Z OK 2022-12-01T10:40:42.8001295Z 2022-12-01T10:40:42.8001380Z Generating XML reports... 2022-12-01T10:40:42.8001764Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221201101822.xml 2022-12-01T10:40:42.8002044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8002168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8002435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8002570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8002757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkgtv_g0 2022-12-01T10:40:42.8002951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkgtv_g0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8003212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8003217Z 2022-12-01T10:40:42.8003289Z Running tests... 2022-12-01T10:40:42.8003484Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8003734Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28285 2022-12-01T10:40:42.8003888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28286 2022-12-01T10:40:42.8004031Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28287 2022-12-01T10:40:42.8004183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28288 2022-12-01T10:40:42.8004458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8004583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8004865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8004999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8005186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm5ccyd6 2022-12-01T10:40:42.8005383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm5ccyd6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8005644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8005767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8006078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8006216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8006400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkc0wivlk 2022-12-01T10:40:42.8006595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkc0wivlk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8006896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8007018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8007297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8007419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8007606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1hkyqgo 2022-12-01T10:40:42.8007859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1hkyqgo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8008222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8008347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8008631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8008770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8008955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg46k07hg 2022-12-01T10:40:42.8009138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg46k07hg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8009303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8009469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8009634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8009798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8009867Z ok (2.139s) 2022-12-01T10:40:42.8009873Z 2022-12-01T10:40:42.8010072Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8010149Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8010155Z 2022-12-01T10:40:42.8010215Z OK 2022-12-01T10:40:42.8010220Z 2022-12-01T10:40:42.8010291Z Generating XML reports... 2022-12-01T10:40:42.8010671Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221201101826.xml 2022-12-01T10:40:42.8010948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8011072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8011354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8011494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8011681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkr7uf98a 2022-12-01T10:40:42.8011882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkr7uf98a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8012122Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8012140Z 2022-12-01T10:40:42.8012200Z Running tests... 2022-12-01T10:40:42.8012396Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8012704Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28528 2022-12-01T10:40:42.8012865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28529 2022-12-01T10:40:42.8013021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28530 2022-12-01T10:40:42.8013176Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28531 2022-12-01T10:40:42.8013453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8013608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8013878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8014014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8014201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vv86lge 2022-12-01T10:40:42.8014399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vv86lge/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8014675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8014797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8015079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8015216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8015401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4zknqyg 2022-12-01T10:40:42.8015583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4zknqyg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8015853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8015979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8016259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8016392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8016572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37eukyc0 2022-12-01T10:40:42.8016765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37eukyc0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8017032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8017141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8017420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8017557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8017739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4irn8io 2022-12-01T10:40:42.8017932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4irn8io/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8018095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8018257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8018422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8018583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8018637Z ok (2.238s) 2022-12-01T10:40:42.8018643Z 2022-12-01T10:40:42.8018838Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8018913Z Ran 1 test in 2.239s 2022-12-01T10:40:42.8018948Z 2022-12-01T10:40:42.8019009Z OK 2022-12-01T10:40:42.8019013Z 2022-12-01T10:40:42.8019098Z Generating XML reports... 2022-12-01T10:40:42.8019476Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221201101829.xml 2022-12-01T10:40:42.8019751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8019902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8020171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8020306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8020491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty_5lb4n 2022-12-01T10:40:42.8020687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty_5lb4n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8020941Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8020946Z 2022-12-01T10:40:42.8021019Z Running tests... 2022-12-01T10:40:42.8021219Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8021444Z test_add_done_callback (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28761 2022-12-01T10:40:42.8021602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28762 2022-12-01T10:40:42.8021744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28763 2022-12-01T10:40:42.8021896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28764 2022-12-01T10:40:42.8022168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8022294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8022573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8022706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8022890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq95_e3lq 2022-12-01T10:40:42.8023082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq95_e3lq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8023344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8023468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8023749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8023886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8024071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn699i3vf 2022-12-01T10:40:42.8024264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn699i3vf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8024535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8024657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8024938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8025058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8025242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3rknk10 2022-12-01T10:40:42.8025436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3rknk10/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8025742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8025866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8026145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8026278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8026492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ci_nfpf 2022-12-01T10:40:42.8026686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ci_nfpf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8026837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8026999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8027165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8027325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8027393Z ok (2.237s) 2022-12-01T10:40:42.8027398Z 2022-12-01T10:40:42.8027596Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8027671Z Ran 1 test in 2.237s 2022-12-01T10:40:42.8027676Z 2022-12-01T10:40:42.8027735Z OK 2022-12-01T10:40:42.8027743Z 2022-12-01T10:40:42.8027814Z Generating XML reports... 2022-12-01T10:40:42.8028164Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101833.xml 2022-12-01T10:40:42.8028435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8028557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8028837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8028971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8029155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gj9qavx 2022-12-01T10:40:42.8029351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gj9qavx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8029607Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8029615Z 2022-12-01T10:40:42.8029677Z Running tests... 2022-12-01T10:40:42.8029873Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8030120Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29000 2022-12-01T10:40:42.8030277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29001 2022-12-01T10:40:42.8030433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29002 2022-12-01T10:40:42.8030587Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29003 2022-12-01T10:40:42.8030860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8030984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8031256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8031390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8031574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wny7gs7 2022-12-01T10:40:42.8031768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wny7gs7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8032099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8032222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8032503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8032637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8032821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnew42ql9 2022-12-01T10:40:42.8033032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnew42ql9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8033303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8033426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8033706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8033841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8034023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79_6kbw6 2022-12-01T10:40:42.8034213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79_6kbw6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8034484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8034595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8034875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8035009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8035192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5el28spv 2022-12-01T10:40:42.8035388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5el28spv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8035553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8035717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8035878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8036042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8036096Z ok (2.240s) 2022-12-01T10:40:42.8036101Z 2022-12-01T10:40:42.8036297Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8036372Z Ran 1 test in 2.240s 2022-12-01T10:40:42.8036377Z 2022-12-01T10:40:42.8036438Z OK 2022-12-01T10:40:42.8036442Z 2022-12-01T10:40:42.8036525Z Generating XML reports... 2022-12-01T10:40:42.8036874Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101837.xml 2022-12-01T10:40:42.8037145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8037271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8037537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8037674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8037857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp818v0mw2 2022-12-01T10:40:42.8038048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp818v0mw2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8038444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8038449Z 2022-12-01T10:40:42.8038523Z Running tests... 2022-12-01T10:40:42.8038786Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8039042Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29227 2022-12-01T10:40:42.8039202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29228 2022-12-01T10:40:42.8039342Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29229 2022-12-01T10:40:42.8039540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29230 2022-12-01T10:40:42.8039818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8039942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8040221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8040359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8040547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkykjvdaj 2022-12-01T10:40:42.8040745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkykjvdaj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8041017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8041129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8041406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8041540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8041722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcyaatvk 2022-12-01T10:40:42.8041919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcyaatvk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8042189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8042309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8042588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8042710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8042897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7kecb_6 2022-12-01T10:40:42.8043094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7kecb_6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8043363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8043487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8043766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8043904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8044089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4m1clf5 2022-12-01T10:40:42.8044282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4m1clf5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8044435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8044598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8044760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8044920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8044988Z ok (2.237s) 2022-12-01T10:40:42.8045031Z 2022-12-01T10:40:42.8045230Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8045304Z Ran 1 test in 2.237s 2022-12-01T10:40:42.8045309Z 2022-12-01T10:40:42.8045370Z OK 2022-12-01T10:40:42.8045375Z 2022-12-01T10:40:42.8045446Z Generating XML reports... 2022-12-01T10:40:42.8045793Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101840.xml 2022-12-01T10:40:42.8046092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8046215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8046493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8046629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8046817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrmxx72o 2022-12-01T10:40:42.8047012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrmxx72o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8047265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8047270Z 2022-12-01T10:40:42.8047330Z Running tests... 2022-12-01T10:40:42.8047528Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8047830Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29454 2022-12-01T10:40:42.8047992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29455 2022-12-01T10:40:42.8048148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29456 2022-12-01T10:40:42.8048303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29457 2022-12-01T10:40:42.8048581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8048706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8048974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8049110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8049297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybmnlsa8 2022-12-01T10:40:42.8049492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybmnlsa8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8049762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8049886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8050171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8050309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8050494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g7cwde5 2022-12-01T10:40:42.8050676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g7cwde5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8050948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8051070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8051346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8051479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8051701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58snm_29 2022-12-01T10:40:42.8051894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58snm_29/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8052164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8052287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8052550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8052714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8052898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehuvplne 2022-12-01T10:40:42.8053091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehuvplne/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8053254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8053417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8053581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8053742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8053797Z ok (2.138s) 2022-12-01T10:40:42.8053802Z 2022-12-01T10:40:42.8054000Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8054077Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8054082Z 2022-12-01T10:40:42.8054140Z OK 2022-12-01T10:40:42.8054144Z 2022-12-01T10:40:42.8054229Z Generating XML reports... 2022-12-01T10:40:42.8054576Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101844.xml 2022-12-01T10:40:42.8054847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8054971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8055238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8055373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8055557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyle7mfxf 2022-12-01T10:40:42.8055755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyle7mfxf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8056008Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8056013Z 2022-12-01T10:40:42.8056085Z Running tests... 2022-12-01T10:40:42.8056281Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8056512Z test_async_function_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29681 2022-12-01T10:40:42.8056671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29682 2022-12-01T10:40:42.8056813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29683 2022-12-01T10:40:42.8056966Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29684 2022-12-01T10:40:42.8057242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8057368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8057651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8057785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8057973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2clzphf1 2022-12-01T10:40:42.8058200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2clzphf1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8058474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8058584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8058866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8059030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8059215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtpc6mti 2022-12-01T10:40:42.8059413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtpc6mti/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8059686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8059811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8060092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8060213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8060396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1neqwwzk 2022-12-01T10:40:42.8060590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1neqwwzk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8060866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8060989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8061267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8061402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8061586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18kdtxch 2022-12-01T10:40:42.8061779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18kdtxch/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8061931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8062094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8062257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8062417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8062485Z ok (2.138s) 2022-12-01T10:40:42.8062492Z 2022-12-01T10:40:42.8062687Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8062763Z Ran 1 test in 2.138s 2022-12-01T10:40:42.8062768Z 2022-12-01T10:40:42.8062827Z OK 2022-12-01T10:40:42.8062834Z 2022-12-01T10:40:42.8062906Z Generating XML reports... 2022-12-01T10:40:42.8063255Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101848.xml 2022-12-01T10:40:42.8063530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8063654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8063939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8064073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8064257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwzzfqmu 2022-12-01T10:40:42.8064454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwzzfqmu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8064736Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8064742Z 2022-12-01T10:40:42.8064802Z Running tests... 2022-12-01T10:40:42.8065000Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8065235Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29904 2022-12-01T10:40:42.8065422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29905 2022-12-01T10:40:42.8065575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29906 2022-12-01T10:40:42.8065728Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29907 2022-12-01T10:40:42.8066005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8066128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8066400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8066536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8066720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrucnmbw 2022-12-01T10:40:42.8066915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrucnmbw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8067194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8067316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8067596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8067731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8067916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yy2o4fz 2022-12-01T10:40:42.8068097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yy2o4fz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8068367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8068490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8068773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8068908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8069090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qgdxrdc 2022-12-01T10:40:42.8069284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qgdxrdc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8069554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8069675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8069941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8070073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8070256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngre73g_ 2022-12-01T10:40:42.8070451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngre73g_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8070616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8070780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8070943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8071131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8071187Z ok (2.243s) 2022-12-01T10:40:42.8071191Z 2022-12-01T10:40:42.8071387Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8071461Z Ran 1 test in 2.243s 2022-12-01T10:40:42.8071466Z 2022-12-01T10:40:42.8071525Z OK 2022-12-01T10:40:42.8071530Z 2022-12-01T10:40:42.8071612Z Generating XML reports... 2022-12-01T10:40:42.8071988Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101851.xml 2022-12-01T10:40:42.8072260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8072384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8072663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8072787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8072970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby_3ubfq 2022-12-01T10:40:42.8073163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby_3ubfq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8073417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8073424Z 2022-12-01T10:40:42.8073497Z Running tests... 2022-12-01T10:40:42.8073691Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8073922Z test_async_function_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30127 2022-12-01T10:40:42.8074080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30128 2022-12-01T10:40:42.8074224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30129 2022-12-01T10:40:42.8074377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30130 2022-12-01T10:40:42.8074656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8074778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8075062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8075200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8075388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocbg0naq 2022-12-01T10:40:42.8075586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocbg0naq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8075857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8075970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8076249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8076385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8076571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnoyao3j 2022-12-01T10:40:42.8076771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnoyao3j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8077041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8077167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8077445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8077600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8077787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnx9brxw0 2022-12-01T10:40:42.8077987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnx9brxw0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8078390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8078575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8078858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8078994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8079177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfj3fceo 2022-12-01T10:40:42.8079375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfj3fceo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8079527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8079691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8079855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8080018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8080088Z ok (2.139s) 2022-12-01T10:40:42.8080093Z 2022-12-01T10:40:42.8080290Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8080366Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8080372Z 2022-12-01T10:40:42.8080433Z OK 2022-12-01T10:40:42.8080437Z 2022-12-01T10:40:42.8080508Z Generating XML reports... 2022-12-01T10:40:42.8080860Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101855.xml 2022-12-01T10:40:42.8081135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8081262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8081542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8081679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8081868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6z5fq2zr 2022-12-01T10:40:42.8082063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6z5fq2zr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8082321Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8082326Z 2022-12-01T10:40:42.8082386Z Running tests... 2022-12-01T10:40:42.8082586Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8082830Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30350 2022-12-01T10:40:42.8082989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30351 2022-12-01T10:40:42.8083144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30352 2022-12-01T10:40:42.8083297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30353 2022-12-01T10:40:42.8083574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8083697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8083978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8084151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8084339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr34lv17y 2022-12-01T10:40:42.8084533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr34lv17y/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8084805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8084926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8085237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8085370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8085554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqosu6b8e 2022-12-01T10:40:42.8085736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqosu6b8e/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8086008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8086132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8086410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8086542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8086728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsbpqy76 2022-12-01T10:40:42.8086924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsbpqy76/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8087197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8087318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8087586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8087775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8087963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptm9h7kwy 2022-12-01T10:40:42.8088158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptm9h7kwy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8088321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8088489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8088650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8088811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8088866Z ok (2.140s) 2022-12-01T10:40:42.8088870Z 2022-12-01T10:40:42.8089072Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8089148Z Ran 1 test in 2.140s 2022-12-01T10:40:42.8089153Z 2022-12-01T10:40:42.8089215Z OK 2022-12-01T10:40:42.8089219Z 2022-12-01T10:40:42.8089303Z Generating XML reports... 2022-12-01T10:40:42.8089653Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101858.xml 2022-12-01T10:40:42.8089925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8090050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8090331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8090454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8090638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8q2kkzv 2022-12-01T10:40:42.8090863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8q2kkzv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8091121Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8091126Z 2022-12-01T10:40:42.8091201Z Running tests... 2022-12-01T10:40:42.8091397Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8091640Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30573 2022-12-01T10:40:42.8091826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30574 2022-12-01T10:40:42.8091968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30575 2022-12-01T10:40:42.8092122Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30576 2022-12-01T10:40:42.8092395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8092519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8092798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8092932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8093116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81p5ixzi 2022-12-01T10:40:42.8093314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81p5ixzi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8093585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8093696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8093979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8094114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8094297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje49qmmq 2022-12-01T10:40:42.8094490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje49qmmq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8094762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8094887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8095164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8095284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8095468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbphvqu03 2022-12-01T10:40:42.8095661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbphvqu03/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8095930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8096053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8096331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8096466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8096649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4w90n7j 2022-12-01T10:40:42.8096841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4w90n7j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8096992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8097186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8097351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8097512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8097580Z ok (2.036s) 2022-12-01T10:40:42.8097585Z 2022-12-01T10:40:42.8097781Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8097887Z Ran 1 test in 2.037s 2022-12-01T10:40:42.8097892Z 2022-12-01T10:40:42.8097951Z OK 2022-12-01T10:40:42.8097956Z 2022-12-01T10:40:42.8098027Z Generating XML reports... 2022-12-01T10:40:42.8098371Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101902.xml 2022-12-01T10:40:42.8098645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8098775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8099053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8099185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8099367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0qi5mm8 2022-12-01T10:40:42.8099560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0qi5mm8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8099815Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8099820Z 2022-12-01T10:40:42.8099880Z Running tests... 2022-12-01T10:40:42.8100077Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8100327Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30796 2022-12-01T10:40:42.8100486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30797 2022-12-01T10:40:42.8100640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30798 2022-12-01T10:40:42.8100794Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30799 2022-12-01T10:40:42.8101065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8101189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8101469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8101590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8101772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_utody2a 2022-12-01T10:40:42.8101966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_utody2a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8102236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8102357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8102637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8102770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8102954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcr1wlxlk 2022-12-01T10:40:42.8103133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcr1wlxlk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8103404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8103523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8103829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8103964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8104149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yx3od9c 2022-12-01T10:40:42.8104342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yx3od9c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8104640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8104762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8105026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8105158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8105341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ffyyfkb 2022-12-01T10:40:42.8105534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ffyyfkb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8105699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8105863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8106026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8106187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8106242Z ok (2.037s) 2022-12-01T10:40:42.8106259Z 2022-12-01T10:40:42.8106445Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8106519Z Ran 1 test in 2.037s 2022-12-01T10:40:42.8106524Z 2022-12-01T10:40:42.8106584Z OK 2022-12-01T10:40:42.8106589Z 2022-12-01T10:40:42.8106675Z Generating XML reports... 2022-12-01T10:40:42.8107025Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101906.xml 2022-12-01T10:40:42.8107297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8107423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8107703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8107828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8108014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3u52ugw 2022-12-01T10:40:42.8108206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3u52ugw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8108463Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8108469Z 2022-12-01T10:40:42.8108540Z Running tests... 2022-12-01T10:40:42.8108735Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8108962Z test_async_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31019 2022-12-01T10:40:42.8109119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31020 2022-12-01T10:40:42.8109264Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31021 2022-12-01T10:40:42.8109419Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31022 2022-12-01T10:40:42.8109694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8109817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8110132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8110269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8110458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmpiqd0n 2022-12-01T10:40:42.8110658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmpiqd0n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8110931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8111070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8111351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8111488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8111677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx93lddip 2022-12-01T10:40:42.8111873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx93lddip/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8112143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8112265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8112546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8112683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8112855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgiwnaxw8 2022-12-01T10:40:42.8113051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgiwnaxw8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8113323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8113449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8113730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8113864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8114046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwfnms80 2022-12-01T10:40:42.8114242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwfnms80/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8114394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8114557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8114717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8114881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8114947Z ok (2.139s) 2022-12-01T10:40:42.8114952Z 2022-12-01T10:40:42.8115149Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8115225Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8115230Z 2022-12-01T10:40:42.8115289Z OK 2022-12-01T10:40:42.8115294Z 2022-12-01T10:40:42.8115364Z Generating XML reports... 2022-12-01T10:40:42.8115714Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101909.xml 2022-12-01T10:40:42.8115987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8116110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8116391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8116553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8116739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t3ngtg7 2022-12-01T10:40:42.8116934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t3ngtg7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8117188Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8117193Z 2022-12-01T10:40:42.8117282Z Running tests... 2022-12-01T10:40:42.8117481Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8117704Z test_async_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31258 2022-12-01T10:40:42.8117859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31259 2022-12-01T10:40:42.8118015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31260 2022-12-01T10:40:42.8118169Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31261 2022-12-01T10:40:42.8118570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8118695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8118975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8119099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8119284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_r7c_8j 2022-12-01T10:40:42.8119478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_r7c_8j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8119752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8119879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8120159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8120293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8120477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_j_dubl 2022-12-01T10:40:42.8120658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_j_dubl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8120931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8121056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8121334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8121467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8121650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_021901_ 2022-12-01T10:40:42.8121839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_021901_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8122112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8122235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8122503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8122634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8122817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqw8i56l 2022-12-01T10:40:42.8123012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqw8i56l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8123222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8123386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8123549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8123710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8123766Z ok (2.140s) 2022-12-01T10:40:42.8123820Z 2022-12-01T10:40:42.8124007Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8124083Z Ran 1 test in 2.140s 2022-12-01T10:40:42.8124088Z 2022-12-01T10:40:42.8124149Z OK 2022-12-01T10:40:42.8124153Z 2022-12-01T10:40:42.8124239Z Generating XML reports... 2022-12-01T10:40:42.8124589Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101913.xml 2022-12-01T10:40:42.8124866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8124990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8125272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8125395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8125581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xs5_5fq 2022-12-01T10:40:42.8125778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xs5_5fq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8126032Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8126037Z 2022-12-01T10:40:42.8126110Z Running tests... 2022-12-01T10:40:42.8126308Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8126551Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31497 2022-12-01T10:40:42.8126707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31498 2022-12-01T10:40:42.8126863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31499 2022-12-01T10:40:42.8127003Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31500 2022-12-01T10:40:42.8127278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8127401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8127736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8127876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8128065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnfhjjow 2022-12-01T10:40:42.8128263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnfhjjow/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8128541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8128651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8128932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8129071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8129255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xuyscbr 2022-12-01T10:40:42.8129449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xuyscbr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8129775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8129898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8130180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8130313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8130483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25u31_wq 2022-12-01T10:40:42.8130732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25u31_wq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8131003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8131125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8131407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8131542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8131728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4aj2ko4 2022-12-01T10:40:42.8131922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4aj2ko4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8132073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8132237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8132400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8132561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8132629Z ok (1.838s) 2022-12-01T10:40:42.8132634Z 2022-12-01T10:40:42.8132833Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8132912Z Ran 1 test in 1.838s 2022-12-01T10:40:42.8132917Z 2022-12-01T10:40:42.8132978Z OK 2022-12-01T10:40:42.8132982Z 2022-12-01T10:40:42.8133066Z Generating XML reports... 2022-12-01T10:40:42.8133402Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101916.xml 2022-12-01T10:40:42.8133675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8133804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8134081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8134216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8134402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwmreaq3 2022-12-01T10:40:42.8134599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwmreaq3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8134856Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8134862Z 2022-12-01T10:40:42.8134921Z Running tests... 2022-12-01T10:40:42.8135120Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8135384Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31568 2022-12-01T10:40:42.8135545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31569 2022-12-01T10:40:42.8135699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31570 2022-12-01T10:40:42.8135852Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31571 2022-12-01T10:40:42.8136154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8136279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8136560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8136682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8136868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprynipf6j 2022-12-01T10:40:42.8137093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprynipf6j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8137363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8137487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8137767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8137904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8138089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9ha7ae5 2022-12-01T10:40:42.8138272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9ha7ae5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8138543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8138670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8138949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8139084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8139268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ermrp5h 2022-12-01T10:40:42.8139465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ermrp5h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8139736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8139859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8140124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8140257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8140443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvh5c7ryk 2022-12-01T10:40:42.8140636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvh5c7ryk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8140799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8140963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8141128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8141288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8141342Z ok (2.142s) 2022-12-01T10:40:42.8141360Z 2022-12-01T10:40:42.8141543Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8141619Z Ran 1 test in 2.142s 2022-12-01T10:40:42.8141626Z 2022-12-01T10:40:42.8141689Z OK 2022-12-01T10:40:42.8141694Z 2022-12-01T10:40:42.8141777Z Generating XML reports... 2022-12-01T10:40:42.8142128Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101919.xml 2022-12-01T10:40:42.8142400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8142523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8142828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8142952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8143136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrm8a9jw 2022-12-01T10:40:42.8143330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrm8a9jw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8143618Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8143624Z 2022-12-01T10:40:42.8143697Z Running tests... 2022-12-01T10:40:42.8143892Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8144125Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31795 2022-12-01T10:40:42.8144286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31796 2022-12-01T10:40:42.8144445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31797 2022-12-01T10:40:42.8144584Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31798 2022-12-01T10:40:42.8144858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8144982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8145270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8145403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8145589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4a3wgv_h 2022-12-01T10:40:42.8145789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4a3wgv_h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8146063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8146174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8146455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8146590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8146775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0b_vxte 2022-12-01T10:40:42.8146967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0b_vxte/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8147236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8147359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8147639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8147773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8147943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3riuerw4 2022-12-01T10:40:42.8148136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3riuerw4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8148405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8148529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8148809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8148942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8149153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdn0di0p 2022-12-01T10:40:42.8149349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdn0di0p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8149500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8149663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8149824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8150012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8150079Z ok (2.144s) 2022-12-01T10:40:42.8150084Z 2022-12-01T10:40:42.8150281Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8150356Z Ran 1 test in 2.144s 2022-12-01T10:40:42.8150361Z 2022-12-01T10:40:42.8150421Z OK 2022-12-01T10:40:42.8150426Z 2022-12-01T10:40:42.8150509Z Generating XML reports... 2022-12-01T10:40:42.8150844Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101923.xml 2022-12-01T10:40:42.8151118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8151243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8151523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8151660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8151845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7wdhehk 2022-12-01T10:40:42.8152039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7wdhehk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8152294Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8152301Z 2022-12-01T10:40:42.8152373Z Running tests... 2022-12-01T10:40:42.8152555Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8152816Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32022 2022-12-01T10:40:42.8152975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32023 2022-12-01T10:40:42.8153132Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32024 2022-12-01T10:40:42.8153285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32025 2022-12-01T10:40:42.8153558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8153682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8153965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8154088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8154272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvth1qg4o 2022-12-01T10:40:42.8154472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvth1qg4o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8154743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8154868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8155145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8155280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8155462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpara8wrhb 2022-12-01T10:40:42.8155683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpara8wrhb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8155942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8156065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8156341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8156507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8156689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s04kl9f 2022-12-01T10:40:42.8156959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8157083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8157361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8157482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8157665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03n5rj4z 2022-12-01T10:40:42.8157858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s04kl9f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8158050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03n5rj4z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8158338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8158504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8158666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8158830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8158898Z ok (2.040s) 2022-12-01T10:40:42.8158904Z 2022-12-01T10:40:42.8159091Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8159167Z Ran 1 test in 2.040s 2022-12-01T10:40:42.8159172Z 2022-12-01T10:40:42.8159234Z OK 2022-12-01T10:40:42.8159239Z 2022-12-01T10:40:42.8159322Z Generating XML reports... 2022-12-01T10:40:42.8159671Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101927.xml 2022-12-01T10:40:42.8159950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8160075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8160357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8160483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8160669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiq00qv8s 2022-12-01T10:40:42.8160864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiq00qv8s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8161121Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8161126Z 2022-12-01T10:40:42.8161199Z Running tests... 2022-12-01T10:40:42.8161397Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8161657Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32249 2022-12-01T10:40:42.8161813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32250 2022-12-01T10:40:42.8161968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32251 2022-12-01T10:40:42.8162158Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32252 2022-12-01T10:40:42.8162439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8162562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8162845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8163016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8163202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa02v50nk 2022-12-01T10:40:42.8163397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa02v50nk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8163669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8163795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8164061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8164196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8164383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhu0kapr 2022-12-01T10:40:42.8164656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8164781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8165065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8165199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8165383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6dn7mpm 2022-12-01T10:40:42.8165567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhu0kapr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8165757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6dn7mpm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8166028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8166149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8166431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8166564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8166744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cnjtvu0 2022-12-01T10:40:42.8166937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cnjtvu0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8167102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8167254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8167414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8167576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8167646Z ok (2.139s) 2022-12-01T10:40:42.8167651Z 2022-12-01T10:40:42.8167913Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8167991Z Ran 1 test in 2.140s 2022-12-01T10:40:42.8167996Z 2022-12-01T10:40:42.8168058Z OK 2022-12-01T10:40:42.8168062Z 2022-12-01T10:40:42.8168145Z Generating XML reports... 2022-12-01T10:40:42.8168480Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101930.xml 2022-12-01T10:40:42.8168796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8168922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8169202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8169341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8169565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmpa1h91 2022-12-01T10:40:42.8169761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmpa1h91/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8170014Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8170020Z 2022-12-01T10:40:42.8170089Z Running tests... 2022-12-01T10:40:42.8170274Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8170500Z test_callback_chain (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32476 2022-12-01T10:40:42.8170658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32477 2022-12-01T10:40:42.8170813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32478 2022-12-01T10:40:42.8170967Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32479 2022-12-01T10:40:42.8171244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8171366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8171652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8171773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8171963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0ktvs3v 2022-12-01T10:40:42.8172158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0ktvs3v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8172429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8172551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8172829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8172965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8173150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdpko243 2022-12-01T10:40:42.8173344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdpko243/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8173605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8173727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8174006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8174140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8174322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3i33itjs 2022-12-01T10:40:42.8174516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3i33itjs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8174786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8174906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8175217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8175352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8175534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jsf16ed 2022-12-01T10:40:42.8175726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jsf16ed/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8175888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8176081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8176241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8176403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8176470Z ok (2.038s) 2022-12-01T10:40:42.8176475Z 2022-12-01T10:40:42.8176661Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8176741Z Ran 1 test in 2.038s 2022-12-01T10:40:42.8176745Z 2022-12-01T10:40:42.8176807Z OK 2022-12-01T10:40:42.8176812Z 2022-12-01T10:40:42.8176896Z Generating XML reports... 2022-12-01T10:40:42.8177247Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101934.xml 2022-12-01T10:40:42.8177521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8177647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8177928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8178049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8178232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptq9vjc1a 2022-12-01T10:40:42.8178429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptq9vjc1a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8178684Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8178689Z 2022-12-01T10:40:42.8178760Z Running tests... 2022-12-01T10:40:42.8178954Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8179180Z test_callback_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32699 2022-12-01T10:40:42.8179338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32700 2022-12-01T10:40:42.8179491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32701 2022-12-01T10:40:42.8179631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32702 2022-12-01T10:40:42.8179902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8180024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8180304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8180436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8180622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkm3n2gly 2022-12-01T10:40:42.8180820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkm3n2gly/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8181093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8181214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8181481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8181642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8181826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptm0gg_9q 2022-12-01T10:40:42.8182021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptm0gg_9q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8182292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8182414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8182720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8182853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8183021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqx_no8y6 2022-12-01T10:40:42.8183212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqx_no8y6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8183484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8183606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8183884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8184017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8184199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38z_pt04 2022-12-01T10:40:42.8184388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38z_pt04/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8184552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8184703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8184866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8185027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8185099Z ok (2.149s) 2022-12-01T10:40:42.8185104Z 2022-12-01T10:40:42.8185303Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8185378Z Ran 1 test in 2.150s 2022-12-01T10:40:42.8185383Z 2022-12-01T10:40:42.8185444Z OK 2022-12-01T10:40:42.8185449Z 2022-12-01T10:40:42.8185534Z Generating XML reports... 2022-12-01T10:40:42.8185870Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101937.xml 2022-12-01T10:40:42.8186142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8186265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8186546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8186679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8186861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfaw8vyi 2022-12-01T10:40:42.8187056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfaw8vyi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8187314Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8187319Z 2022-12-01T10:40:42.8187390Z Running tests... 2022-12-01T10:40:42.8187572Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8187806Z test_callback_with_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 470 2022-12-01T10:40:42.8187962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 471 2022-12-01T10:40:42.8188145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 472 2022-12-01T10:40:42.8188297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 473 2022-12-01T10:40:42.8188571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8188694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8189010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8189132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8189319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6sp9l57 2022-12-01T10:40:42.8189517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6sp9l57/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8189792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8189916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8190196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8190333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8190514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pim5zal 2022-12-01T10:40:42.8190714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pim5zal/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8190972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8191094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8191376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8191510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8191693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgalqibp 2022-12-01T10:40:42.8191893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgalqibp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8192163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8192288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8192553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8192687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8192870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmw27k8z 2022-12-01T10:40:42.8193066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmw27k8z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8193230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8193392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8193555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8193718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8193785Z ok (2.239s) 2022-12-01T10:40:42.8193790Z 2022-12-01T10:40:42.8193977Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8194053Z Ran 1 test in 2.239s 2022-12-01T10:40:42.8194059Z 2022-12-01T10:40:42.8194117Z OK 2022-12-01T10:40:42.8194122Z 2022-12-01T10:40:42.8194207Z Generating XML reports... 2022-12-01T10:40:42.8194585Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101941.xml 2022-12-01T10:40:42.8194859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8194984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8195266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8195419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8195606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptugg39hs 2022-12-01T10:40:42.8195801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptugg39hs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8196054Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8196059Z 2022-12-01T10:40:42.8196131Z Running tests... 2022-12-01T10:40:42.8196331Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8196574Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 711 2022-12-01T10:40:42.8196728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 712 2022-12-01T10:40:42.8196875Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 713 2022-12-01T10:40:42.8197015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 714 2022-12-01T10:40:42.8197288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8197413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8197697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8197834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8198019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4bceujg3 2022-12-01T10:40:42.8198334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4bceujg3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8198612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8198738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8199006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8199141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8199326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffln_gd0 2022-12-01T10:40:42.8199525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffln_gd0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8199802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8199927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8200209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8200345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8200520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjnos91u 2022-12-01T10:40:42.8200716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjnos91u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8200985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8201111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8201437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8201572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8201757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuz54wa51 2022-12-01T10:40:42.8201952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuz54wa51/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8202154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8202305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8202467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8202629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8202698Z ok (2.040s) 2022-12-01T10:40:42.8202706Z 2022-12-01T10:40:42.8202907Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8202984Z Ran 1 test in 2.040s 2022-12-01T10:40:42.8202989Z 2022-12-01T10:40:42.8203050Z OK 2022-12-01T10:40:42.8203055Z 2022-12-01T10:40:42.8203140Z Generating XML reports... 2022-12-01T10:40:42.8203474Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101944.xml 2022-12-01T10:40:42.8203748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8203874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8204154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8204290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8204478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63fgf_1a 2022-12-01T10:40:42.8204673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63fgf_1a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8204927Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8204932Z 2022-12-01T10:40:42.8205047Z Running tests... 2022-12-01T10:40:42.8205230Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8205475Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 934 2022-12-01T10:40:42.8205629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 935 2022-12-01T10:40:42.8205781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 936 2022-12-01T10:40:42.8205931Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 937 2022-12-01T10:40:42.8206210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8206333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8206614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8206737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8206926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd10uzz58 2022-12-01T10:40:42.8207122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd10uzz58/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8207391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8207514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8207887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8208027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8208212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkut3i_f9 2022-12-01T10:40:42.8208405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkut3i_f9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8208695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8208819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8209099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8209232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8209418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mqhpe1l 2022-12-01T10:40:42.8209616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mqhpe1l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8209888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8210011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8210294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8210419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8210602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdmaudie 2022-12-01T10:40:42.8210798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdmaudie/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8210962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8211126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8211287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8211448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8211515Z ok (2.138s) 2022-12-01T10:40:42.8211520Z 2022-12-01T10:40:42.8211703Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8211781Z Ran 1 test in 2.138s 2022-12-01T10:40:42.8211785Z 2022-12-01T10:40:42.8211844Z OK 2022-12-01T10:40:42.8211848Z 2022-12-01T10:40:42.8211933Z Generating XML reports... 2022-12-01T10:40:42.8212280Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101948.xml 2022-12-01T10:40:42.8212553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8212678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8212958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8213092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8213265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4iljui3m 2022-12-01T10:40:42.8213463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4iljui3m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8213721Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8213726Z 2022-12-01T10:40:42.8213797Z Running tests... 2022-12-01T10:40:42.8213993Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8214260Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1157 2022-12-01T10:40:42.8214419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1158 2022-12-01T10:40:42.8214571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1159 2022-12-01T10:40:42.8214711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1160 2022-12-01T10:40:42.8214985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8215138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8215420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8215556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8215742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkb4_39_ 2022-12-01T10:40:42.8215937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkb4_39_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8216210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8216332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8216600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8216738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8216919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdeihrjj4 2022-12-01T10:40:42.8217114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdeihrjj4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8217385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8217510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8217792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8217927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8218097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgq90bbt 2022-12-01T10:40:42.8218372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8218496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8218775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8218909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8219094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75ziceca 2022-12-01T10:40:42.8219289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgq90bbt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8219480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75ziceca/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8219645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8219795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8219958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8220119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8220184Z ok (2.139s) 2022-12-01T10:40:42.8220189Z 2022-12-01T10:40:42.8220387Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8220461Z Ran 1 test in 2.140s 2022-12-01T10:40:42.8220467Z 2022-12-01T10:40:42.8220527Z OK 2022-12-01T10:40:42.8220571Z 2022-12-01T10:40:42.8220657Z Generating XML reports... 2022-12-01T10:40:42.8220996Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101951.xml 2022-12-01T10:40:42.8221272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8221395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8221702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8221837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8222021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpab9ji2ho 2022-12-01T10:40:42.8222218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpab9ji2ho/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8222475Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8222480Z 2022-12-01T10:40:42.8222553Z Running tests... 2022-12-01T10:40:42.8222736Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8223446Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84046 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.220s) 2022-12-01T10:40:42.8223468Z 2022-12-01T10:40:42.8223650Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8223725Z Ran 1 test in 0.220s 2022-12-01T10:40:42.8223729Z 2022-12-01T10:40:42.8223801Z OK (skipped=1) 2022-12-01T10:40:42.8223805Z 2022-12-01T10:40:42.8223890Z Generating XML reports... 2022-12-01T10:40:42.8224240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101955.xml 2022-12-01T10:40:42.8224518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8224641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8224922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8225047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8225233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7u82wlf 2022-12-01T10:40:42.8225437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7u82wlf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8225696Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8225701Z 2022-12-01T10:40:42.8225777Z Running tests... 2022-12-01T10:40:42.8225972Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8226203Z test_future_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1390 2022-12-01T10:40:42.8226358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1391 2022-12-01T10:40:42.8226502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1392 2022-12-01T10:40:42.8226655Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1393 2022-12-01T10:40:42.8226930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8227056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8227367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8227502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8227686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_fnbcs6 2022-12-01T10:40:42.8227881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_fnbcs6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8228140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8228294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8228574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8228709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8228891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_yo3l46 2022-12-01T10:40:42.8229084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_yo3l46/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8229351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8229472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8229750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8229874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8230058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel_bjw7o 2022-12-01T10:40:42.8230253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel_bjw7o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8230523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8230648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8230928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8231065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8231250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjl_p6gr9 2022-12-01T10:40:42.8231432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjl_p6gr9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8231598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8231764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8231928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8232093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8232161Z ok (2.141s) 2022-12-01T10:40:42.8232166Z 2022-12-01T10:40:42.8232366Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8232445Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8232450Z 2022-12-01T10:40:42.8232513Z OK 2022-12-01T10:40:42.8232517Z 2022-12-01T10:40:42.8232589Z Generating XML reports... 2022-12-01T10:40:42.8232936Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101957.xml 2022-12-01T10:40:42.8233214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8233339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8233619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8233811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8233998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpopqo73 2022-12-01T10:40:42.8234197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpopqo73/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8234441Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8234459Z 2022-12-01T10:40:42.8234520Z Running tests... 2022-12-01T10:40:42.8234748Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8234979Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1617 2022-12-01T10:40:42.8235138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1618 2022-12-01T10:40:42.8235293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1619 2022-12-01T10:40:42.8235448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1620 2022-12-01T10:40:42.8235726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8235852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8236119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8236255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8236443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qu7l5h_ 2022-12-01T10:40:42.8236638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qu7l5h_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8236909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8237032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8237315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8237451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8237623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvq2gup7 2022-12-01T10:40:42.8237820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvq2gup7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8238094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8238338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8238622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8238758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8238943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyla_gv02 2022-12-01T10:40:42.8239135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyla_gv02/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8239411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8239521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8239804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8239939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8240120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9f3yynqs 2022-12-01T10:40:42.8240315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9f3yynqs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8240528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8240694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8240858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8241020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8241075Z ok (2.140s) 2022-12-01T10:40:42.8241113Z 2022-12-01T10:40:42.8241310Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8241385Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8241391Z 2022-12-01T10:40:42.8241452Z OK 2022-12-01T10:40:42.8241457Z 2022-12-01T10:40:42.8241542Z Generating XML reports... 2022-12-01T10:40:42.8241893Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102000.xml 2022-12-01T10:40:42.8242172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8242295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8242559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8242693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8242879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcin2jct8 2022-12-01T10:40:42.8243075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcin2jct8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8243328Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8243333Z 2022-12-01T10:40:42.8243406Z Running tests... 2022-12-01T10:40:42.8243601Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8243846Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1844 2022-12-01T10:40:42.8244001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1845 2022-12-01T10:40:42.8244143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1846 2022-12-01T10:40:42.8244295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1847 2022-12-01T10:40:42.8244576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8244700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8244982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8245118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8245305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2c3h42om 2022-12-01T10:40:42.8245500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2c3h42om/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8245761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8245883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8246161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8246299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8246484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bujfch1 2022-12-01T10:40:42.8246682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bujfch1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8246982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8247106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8247382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8247505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8247742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikrqkp0r 2022-12-01T10:40:42.8247982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikrqkp0r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8248259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8248383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8248665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8248807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8248993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxq87zq_v 2022-12-01T10:40:42.8249175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxq87zq_v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8249342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8249508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8249675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8249837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8249907Z ok (2.038s) 2022-12-01T10:40:42.8249912Z 2022-12-01T10:40:42.8250111Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8250186Z Ran 1 test in 2.039s 2022-12-01T10:40:42.8250193Z 2022-12-01T10:40:42.8250254Z OK 2022-12-01T10:40:42.8250258Z 2022-12-01T10:40:42.8250329Z Generating XML reports... 2022-12-01T10:40:42.8250676Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102004.xml 2022-12-01T10:40:42.8250949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8251078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8251358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8251493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8251679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvk4p33hg 2022-12-01T10:40:42.8251875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvk4p33hg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8252118Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8252135Z 2022-12-01T10:40:42.8252194Z Running tests... 2022-12-01T10:40:42.8252390Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8252632Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2067 2022-12-01T10:40:42.8252789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2068 2022-12-01T10:40:42.8252941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2069 2022-12-01T10:40:42.8253092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2070 2022-12-01T10:40:42.8253363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8253516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8253789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8253924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8254108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk3b1gzj 2022-12-01T10:40:42.8254308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk3b1gzj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8254607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8254732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8255013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8255149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8255333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikcd0dzl 2022-12-01T10:40:42.8255516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikcd0dzl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8255790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8255913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8256193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8256326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8256508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dano2kz 2022-12-01T10:40:42.8256702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dano2kz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8256974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8257084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8257365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8257497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8257683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyyhynjr 2022-12-01T10:40:42.8257878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyyhynjr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8258041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8258204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8258367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8258527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8258582Z ok (2.245s) 2022-12-01T10:40:42.8258586Z 2022-12-01T10:40:42.8258782Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8258855Z Ran 1 test in 2.245s 2022-12-01T10:40:42.8258860Z 2022-12-01T10:40:42.8258920Z OK 2022-12-01T10:40:42.8258927Z 2022-12-01T10:40:42.8259011Z Generating XML reports... 2022-12-01T10:40:42.8259361Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102007.xml 2022-12-01T10:40:42.8259636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8259760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8260055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8260190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8260374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptllldt7h 2022-12-01T10:40:42.8260570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptllldt7h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8260827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8260862Z 2022-12-01T10:40:42.8260934Z Running tests... 2022-12-01T10:40:42.8261130Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8261359Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2290 2022-12-01T10:40:42.8261514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2291 2022-12-01T10:40:42.8261657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2292 2022-12-01T10:40:42.8261810Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2293 2022-12-01T10:40:42.8262083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8262206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8262489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8262623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8262810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmj2y52d 2022-12-01T10:40:42.8263007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmj2y52d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8263268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8263392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8263674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8263810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8263995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppylrz2sv 2022-12-01T10:40:42.8264191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppylrz2sv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8264462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8264584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8264865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8264987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8265169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fvj1wwn 2022-12-01T10:40:42.8265364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fvj1wwn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8265637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8265761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8266041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8266176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8266358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cpiv5om 2022-12-01T10:40:42.8266595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cpiv5om/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8266747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8266908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8267070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8267258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8267325Z ok (2.142s) 2022-12-01T10:40:42.8267331Z 2022-12-01T10:40:42.8267527Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8267601Z Ran 1 test in 2.142s 2022-12-01T10:40:42.8267606Z 2022-12-01T10:40:42.8267665Z OK 2022-12-01T10:40:42.8267670Z 2022-12-01T10:40:42.8267740Z Generating XML reports... 2022-12-01T10:40:42.8268094Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102011.xml 2022-12-01T10:40:42.8268367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8268488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8268767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8268902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8269089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpm1l5ou 2022-12-01T10:40:42.8269287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpm1l5ou/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8269541Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8269547Z 2022-12-01T10:40:42.8269606Z Running tests... 2022-12-01T10:40:42.8269805Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8270051Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2513 2022-12-01T10:40:42.8270204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2514 2022-12-01T10:40:42.8270357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2515 2022-12-01T10:40:42.8270512Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2516 2022-12-01T10:40:42.8270788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8270911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8271180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8271318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8271504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij9vaxy5 2022-12-01T10:40:42.8271701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij9vaxy5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8271975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8272101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8272381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8272516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8272700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzgd2v7pe 2022-12-01T10:40:42.8272910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzgd2v7pe/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8273181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8273301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8273579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8273712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8273924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk4ng99f 2022-12-01T10:40:42.8274118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk4ng99f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8274386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8274498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8274781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8274915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8275098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptg2twon9 2022-12-01T10:40:42.8275291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptg2twon9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8275457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8275620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8275780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8275939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8275993Z ok (2.240s) 2022-12-01T10:40:42.8276000Z 2022-12-01T10:40:42.8276197Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8276272Z Ran 1 test in 2.241s 2022-12-01T10:40:42.8276277Z 2022-12-01T10:40:42.8276339Z OK 2022-12-01T10:40:42.8276343Z 2022-12-01T10:40:42.8276426Z Generating XML reports... 2022-12-01T10:40:42.8276776Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102015.xml 2022-12-01T10:40:42.8277051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8277176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8277443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8277577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8277762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42q_zdwa 2022-12-01T10:40:42.8277953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42q_zdwa/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8278324Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8278330Z 2022-12-01T10:40:42.8278405Z Running tests... 2022-12-01T10:40:42.8278603Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8278841Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2736 2022-12-01T10:40:42.8278997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2737 2022-12-01T10:40:42.8279137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2738 2022-12-01T10:40:42.8279291Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2739 2022-12-01T10:40:42.8279626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8279752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8280036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8280174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8280397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2bqissw 2022-12-01T10:40:42.8280595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2bqissw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8280870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8280980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8281267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8281403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8281588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0rdc0wz 2022-12-01T10:40:42.8281786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0rdc0wz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8282060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8282186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8282464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8282587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8282774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmk42ucc3 2022-12-01T10:40:42.8282968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmk42ucc3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8283239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8283364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8283645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8283783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8283968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsul0ibsf 2022-12-01T10:40:42.8284162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsul0ibsf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8284314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8284480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8284645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8284806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8284875Z ok (2.238s) 2022-12-01T10:40:42.8284880Z 2022-12-01T10:40:42.8285078Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8285156Z Ran 1 test in 2.238s 2022-12-01T10:40:42.8285161Z 2022-12-01T10:40:42.8285226Z OK 2022-12-01T10:40:42.8285231Z 2022-12-01T10:40:42.8285304Z Generating XML reports... 2022-12-01T10:40:42.8285654Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102018.xml 2022-12-01T10:40:42.8285930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8286087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8286371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8286504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8286691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxyuq6il 2022-12-01T10:40:42.8286919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxyuq6il/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8287172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8287178Z 2022-12-01T10:40:42.8287237Z Running tests... 2022-12-01T10:40:42.8287437Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8287734Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2959 2022-12-01T10:40:42.8287897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2960 2022-12-01T10:40:42.8288054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2961 2022-12-01T10:40:42.8288255Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2962 2022-12-01T10:40:42.8288532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8288659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8288927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8289064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8289251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkzlaeruz 2022-12-01T10:40:42.8289451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkzlaeruz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8289723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8289849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8290130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8290267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8290451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksrmvc9f 2022-12-01T10:40:42.8290633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksrmvc9f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8290905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8291031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8291309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8291444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8291627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqho3b_ah 2022-12-01T10:40:42.8291822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqho3b_ah/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8292096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8292206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8292487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8292661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8292847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefigq0wg 2022-12-01T10:40:42.8293041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefigq0wg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8293206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8293370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8293558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8293721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8293776Z ok (2.239s) 2022-12-01T10:40:42.8293780Z 2022-12-01T10:40:42.8293978Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8294052Z Ran 1 test in 2.239s 2022-12-01T10:40:42.8294058Z 2022-12-01T10:40:42.8294118Z OK 2022-12-01T10:40:42.8294124Z 2022-12-01T10:40:42.8294209Z Generating XML reports... 2022-12-01T10:40:42.8294553Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102022.xml 2022-12-01T10:40:42.8294827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8294952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8295220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8295355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8295538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3u4rn_b0 2022-12-01T10:40:42.8295731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3u4rn_b0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8295990Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8295995Z 2022-12-01T10:40:42.8296069Z Running tests... 2022-12-01T10:40:42.8296266Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8296516Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3186 2022-12-01T10:40:42.8296674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3187 2022-12-01T10:40:42.8296815Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3188 2022-12-01T10:40:42.8296969Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3189 2022-12-01T10:40:42.8297242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8297365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8297645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8297782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8297967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bz_q0pp 2022-12-01T10:40:42.8298161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bz_q0pp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8298435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8298545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8298824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8298958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8299181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod12axn0 2022-12-01T10:40:42.8299376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod12axn0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8299646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8299769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8300081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8300202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8300386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahxtsfr8 2022-12-01T10:40:42.8300580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahxtsfr8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8300852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8300975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8301251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8301384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8301567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8jdy2dl 2022-12-01T10:40:42.8301762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8jdy2dl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8301913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8302074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8302236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8302397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8302464Z ok (2.739s) 2022-12-01T10:40:42.8302469Z 2022-12-01T10:40:42.8302663Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8302739Z Ran 1 test in 2.739s 2022-12-01T10:40:42.8302744Z 2022-12-01T10:40:42.8302805Z OK 2022-12-01T10:40:42.8302809Z 2022-12-01T10:40:42.8302879Z Generating XML reports... 2022-12-01T10:40:42.8303226Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102026.xml 2022-12-01T10:40:42.8303496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8303620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8303900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8304034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8304216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gtuen3j 2022-12-01T10:40:42.8304410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gtuen3j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8304660Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8304668Z 2022-12-01T10:40:42.8304729Z Running tests... 2022-12-01T10:40:42.8304924Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8305167Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3241 2022-12-01T10:40:42.8305322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3242 2022-12-01T10:40:42.8305502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3243 2022-12-01T10:40:42.8305652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3244 2022-12-01T10:40:42.8305930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8306051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8306332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8306482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8306667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplun5kvkh 2022-12-01T10:40:42.8306862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplun5kvkh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8307135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8307260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8307538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8307672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8307855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw52qgjfc 2022-12-01T10:40:42.8308038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw52qgjfc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8308311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8308433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8308711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8308848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8309028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbehgws0 2022-12-01T10:40:42.8309224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbehgws0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8309492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8309617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8309882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8310015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8310198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2ai6p1e 2022-12-01T10:40:42.8310396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2ai6p1e/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8310561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8310723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8310886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8311046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8311103Z ok (2.141s) 2022-12-01T10:40:42.8311110Z 2022-12-01T10:40:42.8311307Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8311382Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8311387Z 2022-12-01T10:40:42.8311448Z OK 2022-12-01T10:40:42.8311452Z 2022-12-01T10:40:42.8311535Z Generating XML reports... 2022-12-01T10:40:42.8311930Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102030.xml 2022-12-01T10:40:42.8312201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8312325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8312602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8312725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8312941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9_2h2kc 2022-12-01T10:40:42.8313132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9_2h2kc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8313390Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8313395Z 2022-12-01T10:40:42.8313468Z Running tests... 2022-12-01T10:40:42.8313665Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8313897Z test_remote_script_module (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3468 2022-12-01T10:40:42.8314054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3469 2022-12-01T10:40:42.8314195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3470 2022-12-01T10:40:42.8314348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3471 2022-12-01T10:40:42.8314625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8314748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8315030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8315168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8315355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2_yxh4t 2022-12-01T10:40:42.8315548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2_yxh4t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8315816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8315925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8316207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8316341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8316524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphb7s_f1_ 2022-12-01T10:40:42.8316715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphb7s_f1_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8316987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8317108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8317384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8317505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8317690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9eg2ft_u 2022-12-01T10:40:42.8317883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9eg2ft_u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8318152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8318395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8318726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8318863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8319050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxqt12g4 2022-12-01T10:40:42.8319245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxqt12g4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8319396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8319593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8319756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8319954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8320056Z ok (2.137s) 2022-12-01T10:40:42.8320061Z 2022-12-01T10:40:42.8320327Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8320433Z Ran 1 test in 2.137s 2022-12-01T10:40:42.8320438Z 2022-12-01T10:40:42.8320528Z OK 2022-12-01T10:40:42.8320533Z 2022-12-01T10:40:42.8320604Z Generating XML reports... 2022-12-01T10:40:42.8320968Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102033.xml 2022-12-01T10:40:42.8321401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8321556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8321868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8322035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8322249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphil9mu03 2022-12-01T10:40:42.8322475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphil9mu03/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8322761Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8322767Z 2022-12-01T10:40:42.8322877Z Running tests... 2022-12-01T10:40:42.8323063Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8323346Z test_remote_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3691 2022-12-01T10:40:42.8323533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3692 2022-12-01T10:40:42.8323717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3693 2022-12-01T10:40:42.8323897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3694 2022-12-01T10:40:42.8324230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8324383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8324702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8324825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8325044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5tlzfyz 2022-12-01T10:40:42.8325294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5tlzfyz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8325596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8325748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8326086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8326263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8326479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkstg4wgh 2022-12-01T10:40:42.8326703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkstg4wgh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8326961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8327144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8327471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8327640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8327945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkj6w6b6 2022-12-01T10:40:42.8328183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkj6w6b6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8328487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8328640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8328906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8329073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8329287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegm2xnqr 2022-12-01T10:40:42.8329535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegm2xnqr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8329732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8329934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8330126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8330314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8330410Z ok (2.140s) 2022-12-01T10:40:42.8330416Z 2022-12-01T10:40:42.8330602Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8330704Z Ran 1 test in 2.140s 2022-12-01T10:40:42.8330711Z 2022-12-01T10:40:42.8330802Z OK 2022-12-01T10:40:42.8330807Z 2022-12-01T10:40:42.8330941Z Generating XML reports... 2022-12-01T10:40:42.8331325Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102037.xml 2022-12-01T10:40:42.8331657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8331813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8332123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8332245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8332461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyrz9lp_ 2022-12-01T10:40:42.8332684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyrz9lp_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8332971Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8332977Z 2022-12-01T10:40:42.8333104Z Running tests... 2022-12-01T10:40:42.8333333Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8333588Z test_remote_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3914 2022-12-01T10:40:42.8333809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3915 2022-12-01T10:40:42.8333995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3916 2022-12-01T10:40:42.8334136Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3917 2022-12-01T10:40:42.8334440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8334597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8334937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8335150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8335363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcls1ur9 2022-12-01T10:40:42.8335586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcls1ur9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8335890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8336001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8336313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8336486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8336702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kwevu2z 2022-12-01T10:40:42.8336926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kwevu2z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8337247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8337398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8337714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8337877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8338050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuue2ogpt 2022-12-01T10:40:42.8338281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuue2ogpt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8338587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8338738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8339046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8339263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8339480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpes17zt6t 2022-12-01T10:40:42.8339703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpes17zt6t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8339904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8340056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8340246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8340438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8340538Z ok (2.137s) 2022-12-01T10:40:42.8340543Z 2022-12-01T10:40:42.8340770Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8340913Z Ran 1 test in 2.137s 2022-12-01T10:40:42.8340919Z 2022-12-01T10:40:42.8341007Z OK 2022-12-01T10:40:42.8341012Z 2022-12-01T10:40:42.8341140Z Generating XML reports... 2022-12-01T10:40:42.8341508Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102040.xml 2022-12-01T10:40:42.8341811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8341963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8342272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8342493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8342708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyew08ukg 2022-12-01T10:40:42.8342964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyew08ukg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8343251Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8343259Z 2022-12-01T10:40:42.8343362Z Running tests... 2022-12-01T10:40:42.8343545Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8343831Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4153 2022-12-01T10:40:42.8344019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4154 2022-12-01T10:40:42.8344205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4155 2022-12-01T10:40:42.8361706Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4156 2022-12-01T10:40:42.8362108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8362228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8362517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8362647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8362827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjq1i8g01 2022-12-01T10:40:42.8363017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjq1i8g01/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8363287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8363407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8363681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8363808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8363988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6tq6upc 2022-12-01T10:40:42.8364180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6tq6upc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8364442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8364558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8364828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8364956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8365133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdanomns8 2022-12-01T10:40:42.8365322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdanomns8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8365584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8365832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8366108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8366229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8366406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw_lrmo1 2022-12-01T10:40:42.8366639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw_lrmo1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8366797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8366952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8367108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8367263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8367324Z ok (2.040s) 2022-12-01T10:40:42.8367333Z 2022-12-01T10:40:42.8367526Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8367593Z Ran 1 test in 2.040s 2022-12-01T10:40:42.8367599Z 2022-12-01T10:40:42.8367651Z OK 2022-12-01T10:40:42.8367656Z 2022-12-01T10:40:42.8367809Z Generating XML reports... 2022-12-01T10:40:42.8368160Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102044.xml 2022-12-01T10:40:42.8368433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8368549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8368821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8368951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8369125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzs315n5 2022-12-01T10:40:42.8369314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzs315n5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8369561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8369567Z 2022-12-01T10:40:42.8369635Z Running tests... 2022-12-01T10:40:42.8369825Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8370079Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4380 2022-12-01T10:40:42.8370228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4381 2022-12-01T10:40:42.8370374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4382 2022-12-01T10:40:42.8370520Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4383 2022-12-01T10:40:42.8370784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8370900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8371171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8371302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8371481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya7n7jmk 2022-12-01T10:40:42.8371669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya7n7jmk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8371932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8372099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8372369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8372496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8372674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufcmjgni 2022-12-01T10:40:42.8372860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufcmjgni/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8373156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8373272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8373546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8373676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8373852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf4w4fo4 2022-12-01T10:40:42.8374033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf4w4fo4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8374295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8374411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8374682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8374808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8374981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_fh37eof 2022-12-01T10:40:42.8375165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_fh37eof/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8375323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8375476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8375629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8375782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8375845Z ok (2.141s) 2022-12-01T10:40:42.8375850Z 2022-12-01T10:40:42.8376042Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8376110Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8376115Z 2022-12-01T10:40:42.8376166Z OK 2022-12-01T10:40:42.8376170Z 2022-12-01T10:40:42.8376246Z Generating XML reports... 2022-12-01T10:40:42.8376585Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102048.xml 2022-12-01T10:40:42.8376847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8376965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8377238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8377366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8377545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__0tnyv_ 2022-12-01T10:40:42.8377730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__0tnyv_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8377978Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8377984Z 2022-12-01T10:40:42.8378048Z Running tests... 2022-12-01T10:40:42.8378235Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8378485Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4607 2022-12-01T10:40:42.8378632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4608 2022-12-01T10:40:42.8378778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4609 2022-12-01T10:40:42.8378922Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4610 2022-12-01T10:40:42.8379217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8379331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8379607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8379734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8379909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzz0mqls 2022-12-01T10:40:42.8380104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzz0mqls/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8380379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8380505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8380788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8380925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8381110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yy_vu59 2022-12-01T10:40:42.8381304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yy_vu59/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8381564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8381688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8381969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8382106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8382293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68fm1v31 2022-12-01T10:40:42.8382489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68fm1v31/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8382763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8382887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8383170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8383291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8383477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmdws7fz 2022-12-01T10:40:42.8383673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmdws7fz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8383839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8384006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8384171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8384333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8384402Z ok (2.138s) 2022-12-01T10:40:42.8384408Z 2022-12-01T10:40:42.8384607Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8384698Z Ran 1 test in 2.138s 2022-12-01T10:40:42.8384704Z 2022-12-01T10:40:42.8384766Z OK 2022-12-01T10:40:42.8384771Z 2022-12-01T10:40:42.8384857Z Generating XML reports... 2022-12-01T10:40:42.8385207Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102051.xml 2022-12-01T10:40:42.8385483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8385638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8385920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8386059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8386231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowdrtzbj 2022-12-01T10:40:42.8386432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowdrtzbj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8386687Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8386692Z 2022-12-01T10:40:42.8386767Z Running tests... 2022-12-01T10:40:42.8386967Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8387211Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4834 2022-12-01T10:40:42.8387370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4835 2022-12-01T10:40:42.8387525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4836 2022-12-01T10:40:42.8387678Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4837 2022-12-01T10:40:42.8387940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8388065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8388346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8388483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8388671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8i90usn 2022-12-01T10:40:42.8388868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8i90usn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8389144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8389269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8389536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8389675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8389861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpor5bwcod 2022-12-01T10:40:42.8390059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpor5bwcod/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8390330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8390456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8390737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8390873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8391059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsuyetp6 2022-12-01T10:40:42.8391271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsuyetp6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8391544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8391667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8391947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8392081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8392294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdytkns8u 2022-12-01T10:40:42.8392494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdytkns8u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8392660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8392811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8392978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8393137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8393209Z ok (2.342s) 2022-12-01T10:40:42.8393215Z 2022-12-01T10:40:42.8393415Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8393492Z Ran 1 test in 2.343s 2022-12-01T10:40:42.8393499Z 2022-12-01T10:40:42.8393561Z OK 2022-12-01T10:40:42.8393566Z 2022-12-01T10:40:42.8393652Z Generating XML reports... 2022-12-01T10:40:42.8394002Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102055.xml 2022-12-01T10:40:42.8394262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8394388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8394670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8394806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8394991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp527rrmtz 2022-12-01T10:40:42.8395188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp527rrmtz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8395446Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8395451Z 2022-12-01T10:40:42.8395526Z Running tests... 2022-12-01T10:40:42.8395710Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8395939Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5061 2022-12-01T10:40:42.8396097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5062 2022-12-01T10:40:42.8396252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5063 2022-12-01T10:40:42.8396407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5064 2022-12-01T10:40:42.8396684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8396810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8397096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8397233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8397408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyq565p53 2022-12-01T10:40:42.8397605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyq565p53/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8397905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8398029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8398450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8398587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8398823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogyb54tk 2022-12-01T10:40:42.8399018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogyb54tk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8399293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8399404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8399687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8399824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8400007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpib3j6unw 2022-12-01T10:40:42.8400203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpib3j6unw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8400474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8400600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8400878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8401000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8401184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphb0__s9b 2022-12-01T10:40:42.8401377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphb0__s9b/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8401543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8401708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8401869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8402034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8402103Z ok (2.340s) 2022-12-01T10:40:42.8402109Z 2022-12-01T10:40:42.8402307Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8402371Z Ran 1 test in 2.341s 2022-12-01T10:40:42.8402376Z 2022-12-01T10:40:42.8402437Z OK 2022-12-01T10:40:42.8402442Z 2022-12-01T10:40:42.8402526Z Generating XML reports... 2022-12-01T10:40:42.8402875Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102058.xml 2022-12-01T10:40:42.8403147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8403270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8403549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8403689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8403861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8s2xssc 2022-12-01T10:40:42.8404059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8s2xssc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8404311Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8404316Z 2022-12-01T10:40:42.8404426Z Running tests... 2022-12-01T10:40:42.8404626Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8404846Z test_rref_is_owner (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5284 2022-12-01T10:40:42.8405003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5285 2022-12-01T10:40:42.8405158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5286 2022-12-01T10:40:42.8405346Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5287 2022-12-01T10:40:42.8405605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8405728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8406009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8406146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8406333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jdareho 2022-12-01T10:40:42.8406530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jdareho/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8406802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8406930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8407196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8407333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8407517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzf5cxsc 2022-12-01T10:40:42.8407770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzf5cxsc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8408048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8408173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8408453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8408590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8408775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3oyn1fy_ 2022-12-01T10:40:42.8408954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3oyn1fy_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8409227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8409352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8409632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8409768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8409952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1j0z9d6v 2022-12-01T10:40:42.8410147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1j0z9d6v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8410316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8410466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8410630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8410792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8410862Z ok (2.242s) 2022-12-01T10:40:42.8410901Z 2022-12-01T10:40:42.8411103Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8411180Z Ran 1 test in 2.242s 2022-12-01T10:40:42.8411185Z 2022-12-01T10:40:42.8411248Z OK 2022-12-01T10:40:42.8411253Z 2022-12-01T10:40:42.8411339Z Generating XML reports... 2022-12-01T10:40:42.8411689Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102102.xml 2022-12-01T10:40:42.8411980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8412107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8412388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8412524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8412713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxyxpt2x 2022-12-01T10:40:42.8412912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxyxpt2x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8413169Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8413175Z 2022-12-01T10:40:42.8413249Z Running tests... 2022-12-01T10:40:42.8413436Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8413679Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5507 2022-12-01T10:40:42.8413836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5508 2022-12-01T10:40:42.8413992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5509 2022-12-01T10:40:42.8414146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5510 2022-12-01T10:40:42.8414423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8414550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8414831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8414967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8415144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiltg8_l 2022-12-01T10:40:42.8415341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiltg8_l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8415615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8415737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8416020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8416155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8416342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo8bv4xl 2022-12-01T10:40:42.8416539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo8bv4xl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8416813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8416923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8417201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8417334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8417589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpew943ve2 2022-12-01T10:40:42.8417784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpew943ve2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8418054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8418178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8418459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8418608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8418794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5l71h05j 2022-12-01T10:40:42.8418983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5l71h05j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8419146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8419311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8419474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8419636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8419704Z ok (2.139s) 2022-12-01T10:40:42.8419709Z 2022-12-01T10:40:42.8419910Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8419977Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8419982Z 2022-12-01T10:40:42.8420044Z OK 2022-12-01T10:40:42.8420049Z 2022-12-01T10:40:42.8420137Z Generating XML reports... 2022-12-01T10:40:42.8420487Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102106.xml 2022-12-01T10:40:42.8420761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8420887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8421168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8421304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8421476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pdff5s7 2022-12-01T10:40:42.8421678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pdff5s7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8421933Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8421938Z 2022-12-01T10:40:42.8422013Z Running tests... 2022-12-01T10:40:42.8422207Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8422434Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5730 2022-12-01T10:40:42.8422592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5731 2022-12-01T10:40:42.8422749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5732 2022-12-01T10:40:42.8422900Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5733 2022-12-01T10:40:42.8423160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8423285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8423566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8423701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8423884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bxhvee7 2022-12-01T10:40:42.8424108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bxhvee7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8424382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8424506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8424772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8424935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8425121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1799pic7 2022-12-01T10:40:42.8425314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1799pic7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8425586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8425711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8425991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8426125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8426309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4aah8a59 2022-12-01T10:40:42.8426490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4aah8a59/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8426760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8426882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8427161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8427297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8427480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrbiv2v9 2022-12-01T10:40:42.8427673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrbiv2v9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8427837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8427988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8428153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8428315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8428387Z ok (2.139s) 2022-12-01T10:40:42.8428392Z 2022-12-01T10:40:42.8428589Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8428669Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8428674Z 2022-12-01T10:40:42.8428736Z OK 2022-12-01T10:40:42.8428743Z 2022-12-01T10:40:42.8428830Z Generating XML reports... 2022-12-01T10:40:42.8429174Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102110.xml 2022-12-01T10:40:42.8429432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8429557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8429837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8429971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8430154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8uzybwb 2022-12-01T10:40:42.8430349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8uzybwb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8430632Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8430638Z 2022-12-01T10:40:42.8430713Z Running tests... 2022-12-01T10:40:42.8430897Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8431120Z test_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5953 2022-12-01T10:40:42.8431278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5954 2022-12-01T10:40:42.8431460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5955 2022-12-01T10:40:42.8431618Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5956 2022-12-01T10:40:42.8431893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8432018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8432305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8432444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8432618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpernrkmm6 2022-12-01T10:40:42.8432815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpernrkmm6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8433091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8433215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8433496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8433632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8433820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfc15xv7 2022-12-01T10:40:42.8434018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfc15xv7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8434291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8434403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8434688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8434827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8435012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tqofljd 2022-12-01T10:40:42.8435208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tqofljd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8435482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8435607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8435885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8436006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8436193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3axk4__ 2022-12-01T10:40:42.8436389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3axk4__/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8436555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8436720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8436884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8437077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8437148Z ok (2.139s) 2022-12-01T10:40:42.8437153Z 2022-12-01T10:40:42.8437351Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8437414Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8437419Z 2022-12-01T10:40:42.8437481Z OK 2022-12-01T10:40:42.8437485Z 2022-12-01T10:40:42.8437571Z Generating XML reports... 2022-12-01T10:40:42.8437952Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102113.xml 2022-12-01T10:40:42.8438460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8438588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8438870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8439010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8439181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_obvnp0z 2022-12-01T10:40:42.8439376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_obvnp0z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8439638Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8439646Z 2022-12-01T10:40:42.8439720Z Running tests... 2022-12-01T10:40:42.8439920Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8440152Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6176 2022-12-01T10:40:42.8440309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6177 2022-12-01T10:40:42.8440469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6178 2022-12-01T10:40:42.8440623Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6179 2022-12-01T10:40:42.8440884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8441010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8441295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8441433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8441623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivzuzp9m 2022-12-01T10:40:42.8441821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivzuzp9m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8442094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8442222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8442488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8442625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8442807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmld4mok 2022-12-01T10:40:42.8443005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmld4mok/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8443276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8443400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8443684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8443867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8444053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m6no4b5 2022-12-01T10:40:42.8444231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m6no4b5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8444502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8444674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8444953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8445087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8445274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmpvmzi0 2022-12-01T10:40:42.8445471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmpvmzi0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8445637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8445789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8445953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8446115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8446187Z ok (2.240s) 2022-12-01T10:40:42.8446192Z 2022-12-01T10:40:42.8446392Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8446468Z Ran 1 test in 2.240s 2022-12-01T10:40:42.8446473Z 2022-12-01T10:40:42.8446538Z OK 2022-12-01T10:40:42.8446543Z 2022-12-01T10:40:42.8446629Z Generating XML reports... 2022-12-01T10:40:42.8446979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102117.xml 2022-12-01T10:40:42.8447244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8447370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8447650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8447852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8448041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_m55as3 2022-12-01T10:40:42.8448234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_m55as3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8448494Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8448500Z 2022-12-01T10:40:42.8448575Z Running tests... 2022-12-01T10:40:42.8448763Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8449012Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6399 2022-12-01T10:40:42.8449169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6400 2022-12-01T10:40:42.8449323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6401 2022-12-01T10:40:42.8449475Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6402 2022-12-01T10:40:42.8449756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8449879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8450162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8450299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8450509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnr9vhw9c 2022-12-01T10:40:42.8450711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnr9vhw9c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8450986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8451110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8451421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8451559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8451747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmecvrtn 2022-12-01T10:40:42.8451944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmecvrtn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8452218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8452329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8452609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8452745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8452933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptl8r5n0q 2022-12-01T10:40:42.8453126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptl8r5n0q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8453397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8453521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8453802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8453925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8454111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpt7tqwp 2022-12-01T10:40:42.8454306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpt7tqwp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8454470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8454638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8454800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8454961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8455028Z ok (2.239s) 2022-12-01T10:40:42.8455033Z 2022-12-01T10:40:42.8455236Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8455301Z Ran 1 test in 2.239s 2022-12-01T10:40:42.8455306Z 2022-12-01T10:40:42.8455368Z OK 2022-12-01T10:40:42.8455373Z 2022-12-01T10:40:42.8455458Z Generating XML reports... 2022-12-01T10:40:42.8455809Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102120.xml 2022-12-01T10:40:42.8456083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8456209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8456489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8456625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8456796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwo14sqsl 2022-12-01T10:40:42.8457029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwo14sqsl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8457288Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8457293Z 2022-12-01T10:40:42.8457367Z Running tests... 2022-12-01T10:40:42.8457565Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8457798Z test_torchscript_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6626 2022-12-01T10:40:42.8457982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6627 2022-12-01T10:40:42.8458136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6628 2022-12-01T10:40:42.8458289Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6629 2022-12-01T10:40:42.8458551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8458677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8458954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8459091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8459279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6wn2bum 2022-12-01T10:40:42.8459479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6wn2bum/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8459753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8459877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8460144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8460284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8460468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc50kxubk 2022-12-01T10:40:42.8460661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc50kxubk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8460935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8461061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8461339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8461475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8461659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5h2cvz6k 2022-12-01T10:40:42.8461842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5h2cvz6k/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8462113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8462237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8462514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8462652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8462836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi_j190x 2022-12-01T10:40:42.8463030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi_j190x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8463193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8463383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8463550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8463711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8463780Z ok (2.239s) 2022-12-01T10:40:42.8463786Z 2022-12-01T10:40:42.8463985Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8464062Z Ran 1 test in 2.239s 2022-12-01T10:40:42.8464096Z 2022-12-01T10:40:42.8464158Z OK 2022-12-01T10:40:42.8464162Z 2022-12-01T10:40:42.8464248Z Generating XML reports... 2022-12-01T10:40:42.8464596Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102124.xml 2022-12-01T10:40:42.8464856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8464980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8465262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8465398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8465585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8vnv78f 2022-12-01T10:40:42.8465781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8vnv78f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8466041Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8466046Z 2022-12-01T10:40:42.8466118Z Running tests... 2022-12-01T10:40:42.8466316Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8466546Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6849 2022-12-01T10:40:42.8466708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6850 2022-12-01T10:40:42.8466865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6851 2022-12-01T10:40:42.8467018Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6852 2022-12-01T10:40:42.8467294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8467419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8467702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8467837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8468010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrtj4wqx 2022-12-01T10:40:42.8468207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrtj4wqx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8468480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8468605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8468886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8469019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8469204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpietyx_vc 2022-12-01T10:40:42.8469398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpietyx_vc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8469674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8469782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8470089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8470224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8470407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98s57xdl 2022-12-01T10:40:42.8470598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98s57xdl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8470898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8471024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8471304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8471427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8471613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptm_hpb7 2022-12-01T10:40:42.8471806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptm_hpb7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8471970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8472135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8472298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8472464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8472533Z ok (2.139s) 2022-12-01T10:40:42.8472538Z 2022-12-01T10:40:42.8472741Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8472804Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8472809Z 2022-12-01T10:40:42.8472870Z OK 2022-12-01T10:40:42.8472875Z 2022-12-01T10:40:42.8472960Z Generating XML reports... 2022-12-01T10:40:42.8473315Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102128.xml 2022-12-01T10:40:42.8473590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8473713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8473995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8474134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8474308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1tb9dpn 2022-12-01T10:40:42.8474505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1tb9dpn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8474758Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8474765Z 2022-12-01T10:40:42.8474839Z Running tests... 2022-12-01T10:40:42.8475036Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8475285Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7072 2022-12-01T10:40:42.8475440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7073 2022-12-01T10:40:42.8475595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7074 2022-12-01T10:40:42.8475749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7075 2022-12-01T10:40:42.8476013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8476138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8476448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8476586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8476775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqppr74o 2022-12-01T10:40:42.8476974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqppr74o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8477248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8477402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8477684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8477809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8477994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd55bvnxt 2022-12-01T10:40:42.8478313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd55bvnxt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8478588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8478710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8478992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8479130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8479317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6rk211b 2022-12-01T10:40:42.8479499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6rk211b/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8479770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8479896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8480174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8480310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8480496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnnwyk10 2022-12-01T10:40:42.8480690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnnwyk10/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8480858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8481022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8481172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8481336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8481516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.8481695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.8481870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.8482168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.8482470Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.8482647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.8482947Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.8483280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.8483351Z ok (2.139s) 2022-12-01T10:40:42.8483356Z 2022-12-01T10:40:42.8483556Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8483633Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8483639Z 2022-12-01T10:40:42.8483700Z OK 2022-12-01T10:40:42.8483704Z 2022-12-01T10:40:42.8483790Z Generating XML reports... 2022-12-01T10:40:42.8484183Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102131.xml 2022-12-01T10:40:42.8484460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8484571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8484857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8484992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8485181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxi92_xq 2022-12-01T10:40:42.8485379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxi92_xq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8485639Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8485646Z 2022-12-01T10:40:42.8485721Z Running tests... 2022-12-01T10:40:42.8485920Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8486166Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7307 2022-12-01T10:40:42.8486310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7308 2022-12-01T10:40:42.8486465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7309 2022-12-01T10:40:42.8486620Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7310 2022-12-01T10:40:42.8486898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8487024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8487307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8487443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8487633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjzp26et 2022-12-01T10:40:42.8487887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjzp26et/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8488152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8488277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8488557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8488694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8488879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphto2mixt 2022-12-01T10:40:42.8489076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphto2mixt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8489347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8489473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8489737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8489911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8490096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2c4rt_3r 2022-12-01T10:40:42.8490289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2c4rt_3r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8490561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8490715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8490994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8491129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8491318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm54zrgtu 2022-12-01T10:40:42.8491502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm54zrgtu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8491670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8491834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8491999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8492161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8492232Z ok (2.240s) 2022-12-01T10:40:42.8492237Z 2022-12-01T10:40:42.8492435Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8492512Z Ran 1 test in 2.240s 2022-12-01T10:40:42.8492517Z 2022-12-01T10:40:42.8492564Z OK 2022-12-01T10:40:42.8492569Z 2022-12-01T10:40:42.8492656Z Generating XML reports... 2022-12-01T10:40:42.8493007Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102135.xml 2022-12-01T10:40:42.8493283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8493409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8493690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8493829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8494018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhbqzsrm 2022-12-01T10:40:42.8494217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhbqzsrm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8494458Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8494463Z 2022-12-01T10:40:42.8494538Z Running tests... 2022-12-01T10:40:42.8494738Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8494972Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7530 2022-12-01T10:40:42.8495129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7531 2022-12-01T10:40:42.8495283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7532 2022-12-01T10:40:42.8495435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7533 2022-12-01T10:40:42.8495712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8495823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8496104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8496240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8496455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw91hba5z 2022-12-01T10:40:42.8496653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw91hba5z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8496926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8497049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8497360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8497496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8497667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkmgmaff 2022-12-01T10:40:42.8497865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkmgmaff/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8498134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8498258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8498537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8498675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8498862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuak7yax5 2022-12-01T10:40:42.8499057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuak7yax5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8499313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8499436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8499718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8499853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8500037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pq5em20 2022-12-01T10:40:42.8500230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pq5em20/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8500393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8500561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8500725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8500875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8500944Z ok (2.240s) 2022-12-01T10:40:42.8500949Z 2022-12-01T10:40:42.8501150Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8501228Z Ran 1 test in 2.240s 2022-12-01T10:40:42.8501232Z 2022-12-01T10:40:42.8501294Z OK 2022-12-01T10:40:42.8501299Z 2022-12-01T10:40:42.8501383Z Generating XML reports... 2022-12-01T10:40:42.8501730Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102139.xml 2022-12-01T10:40:42.8502004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8502117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8502400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8502536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8502721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8_v3s6s 2022-12-01T10:40:42.8502945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8_v3s6s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8503204Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8503209Z 2022-12-01T10:40:42.8503283Z Running tests... 2022-12-01T10:40:42.8503484Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8503723Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7753 2022-12-01T10:40:42.8503896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7754 2022-12-01T10:40:42.8504051Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7755 2022-12-01T10:40:42.8504204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7756 2022-12-01T10:40:42.8504480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8504607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8504888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8505024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8505209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzvct__a 2022-12-01T10:40:42.8505408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzvct__a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8505666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8505790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8506072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8506208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8506393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpov0gbq76 2022-12-01T10:40:42.8506589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpov0gbq76/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8506859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8506985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8507249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8507387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8507571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8zhm7dz 2022-12-01T10:40:42.8507770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8zhm7dz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8508044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8508169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8508450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8508588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8508772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87q78zr0 2022-12-01T10:40:42.8508954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87q78zr0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8509119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8509373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8509610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8509773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8509842Z ok (2.239s) 2022-12-01T10:40:42.8509848Z 2022-12-01T10:40:42.8510055Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8510166Z Ran 1 test in 2.240s 2022-12-01T10:40:42.8510171Z 2022-12-01T10:40:42.8510219Z OK 2022-12-01T10:40:42.8510223Z 2022-12-01T10:40:42.8510309Z Generating XML reports... 2022-12-01T10:40:42.8510661Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102142.xml 2022-12-01T10:40:42.8510938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8511066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8511349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8511486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8511674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4gabxtn 2022-12-01T10:40:42.8511873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4gabxtn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8512119Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8512124Z 2022-12-01T10:40:42.8512198Z Running tests... 2022-12-01T10:40:42.8512396Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8512663Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7976 2022-12-01T10:40:42.8512822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7977 2022-12-01T10:40:42.8512975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7978 2022-12-01T10:40:42.8513130Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7979 2022-12-01T10:40:42.8513406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8513518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8513800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8513936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8514124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj96f71x6 2022-12-01T10:40:42.8514321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj96f71x6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8514593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8514718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8514997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8515133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8515305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs_kfg9f 2022-12-01T10:40:42.8515501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs_kfg9f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8515772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8515896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8516202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8516338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8516524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvq317p2m 2022-12-01T10:40:42.8516719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvq317p2m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8517049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8517159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8517436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8517571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8517759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvihd2m_ 2022-12-01T10:40:42.8517957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvihd2m_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8518122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8518424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8518588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8518739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8518817Z 10:21:48 Start training 2022-12-01T10:40:42.8518896Z 10:21:48 PS got 0/3 updates 2022-12-01T10:40:42.8518974Z 10:21:48 PS got 1/3 updates 2022-12-01T10:40:42.8519050Z 10:21:48 PS got 2/3 updates 2022-12-01T10:40:42.8519128Z 10:21:48 PS updated model 2022-12-01T10:40:42.8519202Z 10:21:48 PS got 0/3 updates 2022-12-01T10:40:42.8519266Z 10:21:48 PS got 1/3 updates 2022-12-01T10:40:42.8519340Z 10:21:48 PS got 2/3 updates 2022-12-01T10:40:42.8519417Z 10:21:48 PS updated model 2022-12-01T10:40:42.8519491Z 10:21:48 PS got 0/3 updates 2022-12-01T10:40:42.8519565Z 10:21:48 PS got 1/3 updates 2022-12-01T10:40:42.8519639Z 10:21:48 PS got 2/3 updates 2022-12-01T10:40:42.8519717Z 10:21:48 PS updated model 2022-12-01T10:40:42.8519778Z 10:21:48 PS got 0/3 updates 2022-12-01T10:40:42.8519850Z 10:21:48 PS got 1/3 updates 2022-12-01T10:40:42.8519926Z 10:21:48 PS got 2/3 updates 2022-12-01T10:40:42.8520002Z 10:21:48 PS updated model 2022-12-01T10:40:42.8520077Z 10:21:48 Finish training 2022-12-01T10:40:42.8520180Z 10:21:48 Time spent training: 0.017992834997130558s 2022-12-01T10:40:42.8520274Z 10:21:48 worker1 processing one batch 2022-12-01T10:40:42.8520347Z 10:21:48 worker1 reporting grads 2022-12-01T10:40:42.8520435Z 10:21:48 worker1 got updated model 2022-12-01T10:40:42.8520525Z 10:21:48 worker1 processing one batch 2022-12-01T10:40:42.8520613Z 10:21:48 worker1 reporting grads 2022-12-01T10:40:42.8520701Z 10:21:48 worker1 got updated model 2022-12-01T10:40:42.8520791Z 10:21:48 worker1 processing one batch 2022-12-01T10:40:42.8520863Z 10:21:48 worker1 reporting grads 2022-12-01T10:40:42.8520949Z 10:21:48 worker1 got updated model 2022-12-01T10:40:42.8521039Z 10:21:48 worker1 processing one batch 2022-12-01T10:40:42.8521124Z 10:21:48 worker1 reporting grads 2022-12-01T10:40:42.8521210Z 10:21:48 worker1 got updated model 2022-12-01T10:40:42.8521300Z 10:21:48 worker2 processing one batch 2022-12-01T10:40:42.8521385Z 10:21:48 worker2 reporting grads 2022-12-01T10:40:42.8521457Z 10:21:48 worker2 got updated model 2022-12-01T10:40:42.8521546Z 10:21:48 worker2 processing one batch 2022-12-01T10:40:42.8521630Z 10:21:48 worker2 reporting grads 2022-12-01T10:40:42.8521715Z 10:21:48 worker2 got updated model 2022-12-01T10:40:42.8521803Z 10:21:48 worker2 processing one batch 2022-12-01T10:40:42.8521888Z 10:21:48 worker2 reporting grads 2022-12-01T10:40:42.8522024Z 10:21:48 worker2 got updated model 2022-12-01T10:40:42.8522102Z 10:21:48 worker2 processing one batch 2022-12-01T10:40:42.8522186Z 10:21:48 worker2 reporting grads 2022-12-01T10:40:42.8522271Z 10:21:48 worker2 got updated model 2022-12-01T10:40:42.8522357Z 10:21:48 worker3 processing one batch 2022-12-01T10:40:42.8522444Z 10:21:48 worker3 reporting grads 2022-12-01T10:40:42.8522530Z 10:21:48 worker3 got updated model 2022-12-01T10:40:42.8522619Z 10:21:48 worker3 processing one batch 2022-12-01T10:40:42.8522721Z 10:21:48 worker3 reporting grads 2022-12-01T10:40:42.8522806Z 10:21:48 worker3 got updated model 2022-12-01T10:40:42.8522894Z 10:21:48 worker3 processing one batch 2022-12-01T10:40:42.8522977Z 10:21:48 worker3 reporting grads 2022-12-01T10:40:42.8523061Z 10:21:48 worker3 got updated model 2022-12-01T10:40:42.8523150Z 10:21:48 worker3 processing one batch 2022-12-01T10:40:42.8523233Z 10:21:48 worker3 reporting grads 2022-12-01T10:40:42.8523305Z 10:21:48 worker3 got updated model 2022-12-01T10:40:42.8523377Z ok (2.038s) 2022-12-01T10:40:42.8523384Z 2022-12-01T10:40:42.8523590Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8523667Z Ran 1 test in 2.038s 2022-12-01T10:40:42.8523672Z 2022-12-01T10:40:42.8523733Z OK 2022-12-01T10:40:42.8523737Z 2022-12-01T10:40:42.8523824Z Generating XML reports... 2022-12-01T10:40:42.8524208Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20221201102146.xml 2022-12-01T10:40:42.8524491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8524604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8524888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8525025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8525214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp709tu3mz 2022-12-01T10:40:42.8525414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp709tu3mz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8525667Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8525673Z 2022-12-01T10:40:42.8525748Z Running tests... 2022-12-01T10:40:42.8525948Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8526207Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8199 2022-12-01T10:40:42.8526351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8200 2022-12-01T10:40:42.8526505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8201 2022-12-01T10:40:42.8526660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8202 2022-12-01T10:40:42.8526938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8527063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8527347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8527482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8527671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxv727r8u 2022-12-01T10:40:42.8527908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxv727r8u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8528184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8528308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8528621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8528760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8528946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiiujk9f5 2022-12-01T10:40:42.8529144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiiujk9f5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8529448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8529573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8529842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8529978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8530165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu706gqjp 2022-12-01T10:40:42.8530362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu706gqjp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8530633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8530756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8531036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8531174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8531360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodbrl5qq 2022-12-01T10:40:42.8531542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodbrl5qq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8531711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8531875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8532038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8532202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8532308Z Solved! Running reward is now 489.8051609821906! 2022-12-01T10:40:42.8532381Z ok (2.238s) 2022-12-01T10:40:42.8532387Z 2022-12-01T10:40:42.8532588Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8532651Z Ran 1 test in 2.239s 2022-12-01T10:40:42.8532656Z 2022-12-01T10:40:42.8532718Z OK 2022-12-01T10:40:42.8532723Z 2022-12-01T10:40:42.8532807Z Generating XML reports... 2022-12-01T10:40:42.8533225Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20221201102149.xml 2022-12-01T10:40:42.8533503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8533628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8533910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8534049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8534228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6qhe561 2022-12-01T10:40:42.8534427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6qhe561/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8534684Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8534689Z 2022-12-01T10:40:42.8534765Z Running tests... 2022-12-01T10:40:42.8534963Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8535226Z test_bad_module (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8425 2022-12-01T10:40:42.8535384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8426 2022-12-01T10:40:42.8535659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8535785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8536078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8536217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8536407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9wtjgiq 2022-12-01T10:40:42.8536607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9wtjgiq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8536883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8537008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8537289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8537425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8537614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnj16jqu 2022-12-01T10:40:42.8537800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnj16jqu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8537966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8538130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8538229Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.8538660Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-12-01T10:40:42.8538754Z Traceback (most recent call last): 2022-12-01T10:40:42.8539017Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8539153Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8539413Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module 2022-12-01T10:40:42.8539563Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-12-01T10:40:42.8539986Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-12-01T10:40:42.8539992Z 2022-12-01T10:40:42.8540087Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.8541707Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .")\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module\n "Expect `module_cls(*args, **kwargs)` returns an instance of , "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .\n') 2022-12-01T10:40:42.8541809Z Traceback (most recent call last): 2022-12-01T10:40:42.8542079Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8542214Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8542369Z File "/tmp/tmpd6qhe561/_remote_module_non_scriptable.py", line 47, in _remote_forward 2022-12-01T10:40:42.8542463Z module = module_rref.local_value() 2022-12-01T10:40:42.8542762Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:42.8542990Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:42.8543088Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.8543517Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-12-01T10:40:42.8543612Z Traceback (most recent call last): 2022-12-01T10:40:42.8543876Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8544010Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8544283Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module 2022-12-01T10:40:42.8544433Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-12-01T10:40:42.8544857Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-12-01T10:40:42.8544864Z 2022-12-01T10:40:42.8544869Z 2022-12-01T10:40:42.8544964Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.8545393Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-12-01T10:40:42.8545472Z Traceback (most recent call last): 2022-12-01T10:40:42.8545733Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8545868Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8546141Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module 2022-12-01T10:40:42.8546289Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-12-01T10:40:42.8546709Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-12-01T10:40:42.8546715Z 2022-12-01T10:40:42.8546811Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.8548272Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .")\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module\n "Expect `module_cls(*args, **kwargs)` returns an instance of , "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .\n') 2022-12-01T10:40:42.8548373Z Traceback (most recent call last): 2022-12-01T10:40:42.8548662Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8548797Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8548953Z File "/tmp/tmpd6qhe561/_remote_module_non_scriptable.py", line 47, in _remote_forward 2022-12-01T10:40:42.8549034Z module = module_rref.local_value() 2022-12-01T10:40:42.8549303Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:42.8549559Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:42.8549670Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.8550094Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-12-01T10:40:42.8550191Z Traceback (most recent call last): 2022-12-01T10:40:42.8550450Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8550587Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8550858Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module 2022-12-01T10:40:42.8550991Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-12-01T10:40:42.8551419Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-12-01T10:40:42.8551424Z 2022-12-01T10:40:42.8551429Z 2022-12-01T10:40:42.8551499Z ok (1.834s) 2022-12-01T10:40:42.8551504Z 2022-12-01T10:40:42.8551704Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8551783Z Ran 1 test in 1.834s 2022-12-01T10:40:42.8551788Z 2022-12-01T10:40:42.8551851Z OK 2022-12-01T10:40:42.8551856Z 2022-12-01T10:40:42.8551942Z Generating XML reports... 2022-12-01T10:40:42.8552314Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102153.xml 2022-12-01T10:40:42.8552589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8552719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8552990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8553129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8553317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo4bsnwb 2022-12-01T10:40:42.8553520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo4bsnwb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8553775Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8553781Z 2022-12-01T10:40:42.8553855Z Running tests... 2022-12-01T10:40:42.8554052Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8554289Z test_forward_async (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8542 2022-12-01T10:40:42.8554434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8543 2022-12-01T10:40:42.8554709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8554835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8555116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8555290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8555479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxlwts6o 2022-12-01T10:40:42.8555678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxlwts6o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8555952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8556104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8556373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8556513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8556700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5e34xrq 2022-12-01T10:40:42.8556898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5e34xrq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8557064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8557227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8557504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxlwts6o/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8557779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5e34xrq/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8558061Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpdxlwts6o/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8558117Z ok (1.933s) 2022-12-01T10:40:42.8558136Z 2022-12-01T10:40:42.8558446Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8558523Z Ran 1 test in 1.933s 2022-12-01T10:40:42.8558528Z 2022-12-01T10:40:42.8558589Z OK 2022-12-01T10:40:42.8558594Z 2022-12-01T10:40:42.8558681Z Generating XML reports... 2022-12-01T10:40:42.8559048Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102156.xml 2022-12-01T10:40:42.8559324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8559453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8559737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8559859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8560049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ahn0epp 2022-12-01T10:40:42.8560247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ahn0epp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8560502Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8560508Z 2022-12-01T10:40:42.8560581Z Running tests... 2022-12-01T10:40:42.8560779Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8561022Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8659 2022-12-01T10:40:42.8561179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8660 2022-12-01T10:40:42.8561457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8561569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8561903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8562037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8562221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybu0g0gx 2022-12-01T10:40:42.8562418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybu0g0gx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8562731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8562855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8563136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8563258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8563446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1gczwgt 2022-12-01T10:40:42.8563640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1gczwgt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8563805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8563970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8564247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1gczwgt/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8564521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybu0g0gx/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8564801Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpo1gczwgt/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8564872Z ok (2.033s) 2022-12-01T10:40:42.8564877Z 2022-12-01T10:40:42.8565079Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8565143Z Ran 1 test in 2.034s 2022-12-01T10:40:42.8565148Z 2022-12-01T10:40:42.8565208Z OK 2022-12-01T10:40:42.8565213Z 2022-12-01T10:40:42.8565300Z Generating XML reports... 2022-12-01T10:40:42.8565670Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102200.xml 2022-12-01T10:40:42.8565949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8566073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8566356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8566495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8566671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyyn5jvx 2022-12-01T10:40:42.8566871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyyn5jvx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8567128Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8567134Z 2022-12-01T10:40:42.8567209Z Running tests... 2022-12-01T10:40:42.8567407Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8567642Z test_forward_sync (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8780 2022-12-01T10:40:42.8567880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8781 2022-12-01T10:40:42.8568162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8568326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8568598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8568736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8568921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87_dxrvh 2022-12-01T10:40:42.8569122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87_dxrvh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8569425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8569550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8569832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8569972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8570158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmlqdoca 2022-12-01T10:40:42.8570344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmlqdoca/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8570509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8570673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8570949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmlqdoca/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8571218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87_dxrvh/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8571501Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpnmlqdoca/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8571575Z ok (1.933s) 2022-12-01T10:40:42.8571580Z 2022-12-01T10:40:42.8571779Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8571857Z Ran 1 test in 1.933s 2022-12-01T10:40:42.8571862Z 2022-12-01T10:40:42.8571909Z OK 2022-12-01T10:40:42.8571914Z 2022-12-01T10:40:42.8572001Z Generating XML reports... 2022-12-01T10:40:42.8572372Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102203.xml 2022-12-01T10:40:42.8572647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8572772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8573052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8573189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8573376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvk44q73v 2022-12-01T10:40:42.8573572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvk44q73v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8573814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8573821Z 2022-12-01T10:40:42.8573896Z Running tests... 2022-12-01T10:40:42.8574094Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8574337Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8901 2022-12-01T10:40:42.8574494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8902 2022-12-01T10:40:42.8574808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8574934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8575216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8575339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8575527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv18d2lev 2022-12-01T10:40:42.8575748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv18d2lev/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8576022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8576146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8576427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8576563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8576749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkkn61gb 2022-12-01T10:40:42.8576945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkkn61gb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8577097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8577264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8577538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv18d2lev/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8577811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkkn61gb/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8578091Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpfkkn61gb/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8578161Z ok (1.933s) 2022-12-01T10:40:42.8578166Z 2022-12-01T10:40:42.8578365Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8578442Z Ran 1 test in 1.934s 2022-12-01T10:40:42.8578447Z 2022-12-01T10:40:42.8578511Z OK 2022-12-01T10:40:42.8578517Z 2022-12-01T10:40:42.8578587Z Generating XML reports... 2022-12-01T10:40:42.8578959Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102207.xml 2022-12-01T10:40:42.8579234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8579358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8579642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8579780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8579968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomqri029 2022-12-01T10:40:42.8580163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomqri029/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8580419Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8580425Z 2022-12-01T10:40:42.8580485Z Running tests... 2022-12-01T10:40:42.8580679Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8580917Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9022 2022-12-01T10:40:42.8581106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9023 2022-12-01T10:40:42.8581384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8581508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8581790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8581925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8582139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3owtdar 2022-12-01T10:40:42.8582321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3owtdar/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8582595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8582719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8583004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8583141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8583326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf6isu60 2022-12-01T10:40:42.8583522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf6isu60/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8583688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8583839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8583908Z ok (1.833s) 2022-12-01T10:40:42.8583913Z 2022-12-01T10:40:42.8584109Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8584187Z Ran 1 test in 1.834s 2022-12-01T10:40:42.8584192Z 2022-12-01T10:40:42.8584254Z OK 2022-12-01T10:40:42.8584261Z 2022-12-01T10:40:42.8584345Z Generating XML reports... 2022-12-01T10:40:42.8584711Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102210.xml 2022-12-01T10:40:42.8584982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8585105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8585374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8585510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8585694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8f6yo89 2022-12-01T10:40:42.8585890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8f6yo89/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8586145Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8586150Z 2022-12-01T10:40:42.8586224Z Running tests... 2022-12-01T10:40:42.8586421Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8586661Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9139 2022-12-01T10:40:42.8586804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9140 2022-12-01T10:40:42.8587085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8587211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8587492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8587631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8587854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposrsqbpr 2022-12-01T10:40:42.8588055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposrsqbpr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8588329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8588454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8588748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8588885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8589072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09lltji7 2022-12-01T10:40:42.8589266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09lltji7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8589434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8589601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8589671Z ok (1.834s) 2022-12-01T10:40:42.8589676Z 2022-12-01T10:40:42.8589873Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8589936Z Ran 1 test in 1.834s 2022-12-01T10:40:42.8589956Z 2022-12-01T10:40:42.8590006Z OK 2022-12-01T10:40:42.8590011Z 2022-12-01T10:40:42.8590092Z Generating XML reports... 2022-12-01T10:40:42.8590457Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102213.xml 2022-12-01T10:40:42.8590729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8590854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8591140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8591279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8591467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip1ylmo1 2022-12-01T10:40:42.8591651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip1ylmo1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8591907Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8591912Z 2022-12-01T10:40:42.8591988Z Running tests... 2022-12-01T10:40:42.8592185Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8592444Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9256 2022-12-01T10:40:42.8592604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9257 2022-12-01T10:40:42.8592887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8593013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8593300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8593423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8593614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcnnbplc 2022-12-01T10:40:42.8593812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcnnbplc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8594084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8594207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8594528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8594665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8594851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4_ak88q 2022-12-01T10:40:42.8595030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4_ak88q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8595224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8595388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8595458Z ok (1.832s) 2022-12-01T10:40:42.8595463Z 2022-12-01T10:40:42.8595661Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8595738Z Ran 1 test in 1.833s 2022-12-01T10:40:42.8595742Z 2022-12-01T10:40:42.8595805Z OK 2022-12-01T10:40:42.8595811Z 2022-12-01T10:40:42.8595898Z Generating XML reports... 2022-12-01T10:40:42.8596265Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102217.xml 2022-12-01T10:40:42.8596526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8596653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8596938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8597075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8597263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbotyb96 2022-12-01T10:40:42.8597458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbotyb96/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8597718Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8597724Z 2022-12-01T10:40:42.8597799Z Running tests... 2022-12-01T10:40:42.8597982Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8598384Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9373 2022-12-01T10:40:42.8598548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9374 2022-12-01T10:40:42.8598825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8598950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8599235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8599376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8599566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6nbj_9_ 2022-12-01T10:40:42.8599763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6nbj_9_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8600026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8600155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8600439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8600575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8600761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gm6wvel 2022-12-01T10:40:42.8600960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gm6wvel/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8601208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8601377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8601655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gm6wvel/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8601953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6nbj_9_/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8602233Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpf6nbj_9_/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:42.8602302Z ok (1.834s) 2022-12-01T10:40:42.8602307Z 2022-12-01T10:40:42.8602509Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8602587Z Ran 1 test in 1.834s 2022-12-01T10:40:42.8602592Z 2022-12-01T10:40:42.8602654Z OK 2022-12-01T10:40:42.8602659Z 2022-12-01T10:40:42.8602744Z Generating XML reports... 2022-12-01T10:40:42.8603112Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102220.xml 2022-12-01T10:40:42.8603389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8603503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8603789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8603926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8604113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxev88cri 2022-12-01T10:40:42.8604316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxev88cri/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8604571Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8604576Z 2022-12-01T10:40:42.8604652Z Running tests... 2022-12-01T10:40:42.8604850Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8605092Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9490 2022-12-01T10:40:42.8605238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9491 2022-12-01T10:40:42.8605513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8605637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8605920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8606056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8606242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yc2x34w 2022-12-01T10:40:42.8606437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yc2x34w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8606709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8606838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8607105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8607242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8607428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdswat5rr 2022-12-01T10:40:42.8607666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdswat5rr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8607896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8608060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8608132Z ok (1.832s) 2022-12-01T10:40:42.8608137Z 2022-12-01T10:40:42.8608342Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8608447Z Ran 1 test in 1.833s 2022-12-01T10:40:42.8608452Z 2022-12-01T10:40:42.8608518Z OK 2022-12-01T10:40:42.8608523Z 2022-12-01T10:40:42.8608610Z Generating XML reports... 2022-12-01T10:40:42.8608982Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102223.xml 2022-12-01T10:40:42.8609261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8609387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8609670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8609809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8609998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbclwkyzl 2022-12-01T10:40:42.8610185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbclwkyzl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8610440Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8610445Z 2022-12-01T10:40:42.8610520Z Running tests... 2022-12-01T10:40:42.8610719Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8611005Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9607 2022-12-01T10:40:42.8611163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9608 2022-12-01T10:40:42.8611441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8611568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8611855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8611977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8612165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sowrgdp 2022-12-01T10:40:42.8612364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sowrgdp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8612642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8612770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8613051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8613189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8613377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a14g8_p 2022-12-01T10:40:42.8613562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a14g8_p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8613728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8613893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8614283Z The new attribute ``new_attr`` of RemoteModule is ignored during RPC pickling. To pickle this attribute, please add it to ``_REMOTE_MODULE_PICKLED_ATTRIBUTES``. Otherwise, please explicitly add it to ``_REMOTE_MODULE_ATTRIBUTES_IGNORE_FOR_PICKLING``. 2022-12-01T10:40:42.8614356Z ok (1.833s) 2022-12-01T10:40:42.8614362Z 2022-12-01T10:40:42.8614562Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8614641Z Ran 1 test in 1.833s 2022-12-01T10:40:42.8614647Z 2022-12-01T10:40:42.8614710Z OK 2022-12-01T10:40:42.8614741Z 2022-12-01T10:40:42.8614828Z Generating XML reports... 2022-12-01T10:40:42.8615185Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102226.xml 2022-12-01T10:40:42.8615460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8615585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8615871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8616011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8616199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lcsu1r0 2022-12-01T10:40:42.8616396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lcsu1r0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8616655Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8616663Z 2022-12-01T10:40:42.8616737Z Running tests... 2022-12-01T10:40:42.8616921Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8617152Z test_train_eval (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9724 2022-12-01T10:40:42.8617309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9725 2022-12-01T10:40:42.8617588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8617714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8617995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8618133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8618323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiwloyay 2022-12-01T10:40:42.8618521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiwloyay/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8618780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8618905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8619190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8619326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8619512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2g3ueq_c 2022-12-01T10:40:42.8619712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2g3ueq_c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8619878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8620044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8620100Z ok (1.836s) 2022-12-01T10:40:42.8620105Z 2022-12-01T10:40:42.8620304Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8620380Z Ran 1 test in 1.836s 2022-12-01T10:40:42.8620386Z 2022-12-01T10:40:42.8620448Z OK 2022-12-01T10:40:42.8620453Z 2022-12-01T10:40:42.8620538Z Generating XML reports... 2022-12-01T10:40:42.8620970Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102230.xml 2022-12-01T10:40:42.8621246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8621372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8621656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8621809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8621999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yzabvgc 2022-12-01T10:40:42.8622196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yzabvgc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8622453Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8622458Z 2022-12-01T10:40:42.8622536Z Running tests... 2022-12-01T10:40:42.8622734Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8622980Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9841 2022-12-01T10:40:42.8623138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9842 2022-12-01T10:40:42.8623401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8623526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8623806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8623941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8624130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdkp8ryr 2022-12-01T10:40:42.8624327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdkp8ryr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8624606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8624731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8625011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8625136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8625321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3nvbolx 2022-12-01T10:40:42.8625516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3nvbolx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8625682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8625848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8625918Z ok (1.837s) 2022-12-01T10:40:42.8625923Z 2022-12-01T10:40:42.8626122Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8626203Z Ran 1 test in 1.838s 2022-12-01T10:40:42.8626208Z 2022-12-01T10:40:42.8626256Z OK 2022-12-01T10:40:42.8626274Z 2022-12-01T10:40:42.8626349Z Generating XML reports... 2022-12-01T10:40:42.8626717Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102233.xml 2022-12-01T10:40:42.8626991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8627116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8627426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8627562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8627749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7u1yxie7 2022-12-01T10:40:42.8627945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7u1yxie7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8628187Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8628221Z 2022-12-01T10:40:42.8628297Z Running tests... 2022-12-01T10:40:42.8628498Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8628705Z test_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9958 2022-12-01T10:40:42.8628863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9959 2022-12-01T10:40:42.8629018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9960 2022-12-01T10:40:42.8629173Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9961 2022-12-01T10:40:42.8629446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8629555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8629839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8629978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8630164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0ifmyx6 2022-12-01T10:40:42.8630365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0ifmyx6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8630640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8630764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8631048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8631186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8631357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxjxbhhh 2022-12-01T10:40:42.8631562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxjxbhhh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8631832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8631956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8632234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8632372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8632557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngril484 2022-12-01T10:40:42.8632751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngril484/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8633023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8633136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8633416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8633550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8633734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezbeypdv 2022-12-01T10:40:42.8633971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezbeypdv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8634137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8634303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8634465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8634616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8634714Z ok (2.038s) 2022-12-01T10:40:42.8634719Z 2022-12-01T10:40:42.8634918Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8634995Z Ran 1 test in 2.038s 2022-12-01T10:40:42.8635000Z 2022-12-01T10:40:42.8635062Z OK 2022-12-01T10:40:42.8635066Z 2022-12-01T10:40:42.8635152Z Generating XML reports... 2022-12-01T10:40:42.8635496Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102236.xml 2022-12-01T10:40:42.8635769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8635893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8636159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8636294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8636481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqr8ju80 2022-12-01T10:40:42.8636675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqr8ju80/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8636928Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8636934Z 2022-12-01T10:40:42.8637009Z Running tests... 2022-12-01T10:40:42.8637208Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8637434Z test_add_done_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10181 2022-12-01T10:40:42.8637578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10182 2022-12-01T10:40:42.8637735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10183 2022-12-01T10:40:42.8637891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10184 2022-12-01T10:40:42.8638167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8638477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8638757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8638892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8639078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmchv8ie0 2022-12-01T10:40:42.8639274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmchv8ie0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8639531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8639655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8639936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8640070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8640254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88p5r76h 2022-12-01T10:40:42.8640448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88p5r76h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8640800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8640925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8641191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8641325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8641544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusxe5w8k 2022-12-01T10:40:42.8641739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusxe5w8k/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8642009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8642132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8642413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8642549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8642734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bip79_p 2022-12-01T10:40:42.8642916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bip79_p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8643081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8643248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8643409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8643572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8643640Z ok (2.040s) 2022-12-01T10:40:42.8643645Z 2022-12-01T10:40:42.8643844Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8643921Z Ran 1 test in 2.040s 2022-12-01T10:40:42.8643926Z 2022-12-01T10:40:42.8643973Z OK 2022-12-01T10:40:42.8643977Z 2022-12-01T10:40:42.8644064Z Generating XML reports... 2022-12-01T10:40:42.8644407Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102240.xml 2022-12-01T10:40:42.8644678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8644805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8645082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8645217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8645401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0dyqc50 2022-12-01T10:40:42.8645598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0dyqc50/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8645840Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8645845Z 2022-12-01T10:40:42.8645921Z Running tests... 2022-12-01T10:40:42.8646118Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8646335Z test_add_with_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10404 2022-12-01T10:40:42.8646495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10405 2022-12-01T10:40:42.8646652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10406 2022-12-01T10:40:42.8646807Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10407 2022-12-01T10:40:42.8647122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8647233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8647518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8647655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8647911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlpv59u4 2022-12-01T10:40:42.8648151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlpv59u4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8648431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8648558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8648836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8648975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8649145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp395163at 2022-12-01T10:40:42.8649338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp395163at/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8649607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8649735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8650012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8650147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8650331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwg04ztbv 2022-12-01T10:40:42.8650531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwg04ztbv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8650785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8650908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8651186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8651325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8651509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjy0uws79 2022-12-01T10:40:42.8651702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjy0uws79/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8651869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8652037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8652200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8652364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8652454Z ok (2.141s) 2022-12-01T10:40:42.8652461Z 2022-12-01T10:40:42.8652747Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8652851Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8652859Z 2022-12-01T10:40:42.8652922Z OK 2022-12-01T10:40:42.8652926Z 2022-12-01T10:40:42.8653015Z Generating XML reports... 2022-12-01T10:40:42.8653361Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102243.xml 2022-12-01T10:40:42.8653635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8653748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8654078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8654217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8654405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4j2gu0v 2022-12-01T10:40:42.8654603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4j2gu0v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8654889Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8654894Z 2022-12-01T10:40:42.8654971Z Running tests... 2022-12-01T10:40:42.8655170Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8655387Z test_all_gather (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10627 2022-12-01T10:40:42.8655533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10628 2022-12-01T10:40:42.8655690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10629 2022-12-01T10:40:42.8655842Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10630 2022-12-01T10:40:42.8656117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8656240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8656528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8656666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8656855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7422w1o 2022-12-01T10:40:42.8657037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7422w1o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8657313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8657438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8657717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8657856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8658045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso9les6r 2022-12-01T10:40:42.8658243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso9les6r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8658515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8658642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8658911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8659048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8659233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqp1_4n5z 2022-12-01T10:40:42.8659430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqp1_4n5z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8659703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8659830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8660110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8660247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8660464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx35nsgv 2022-12-01T10:40:42.8660648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx35nsgv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8660815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8660980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8661144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8661338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8661407Z ok (2.039s) 2022-12-01T10:40:42.8661413Z 2022-12-01T10:40:42.8661614Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8661690Z Ran 1 test in 2.039s 2022-12-01T10:40:42.8661695Z 2022-12-01T10:40:42.8661743Z OK 2022-12-01T10:40:42.8661747Z 2022-12-01T10:40:42.8661833Z Generating XML reports... 2022-12-01T10:40:42.8662178Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102247.xml 2022-12-01T10:40:42.8662453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8662579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8662860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8663002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8663190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8e586cs 2022-12-01T10:40:42.8663386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8e586cs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8663706Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8663712Z 2022-12-01T10:40:42.8663789Z Running tests... 2022-12-01T10:40:42.8663989Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8664214Z test_all_gather_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10850 2022-12-01T10:40:42.8664372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10851 2022-12-01T10:40:42.8664529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10852 2022-12-01T10:40:42.8664686Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10853 2022-12-01T10:40:42.8664960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8665070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8665357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8665494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8665682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdb6kbcmn 2022-12-01T10:40:42.8665880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdb6kbcmn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8666152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8666278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8666560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8666696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8666868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnfj0q_2 2022-12-01T10:40:42.8667115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnfj0q_2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8667390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8667514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8667793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8667958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8668143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tvvnkc6 2022-12-01T10:40:42.8668339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tvvnkc6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8668596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8668747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8669026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8669162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8669347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ajb2anm 2022-12-01T10:40:42.8669542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ajb2anm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8669709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8669874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8670040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8670188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8670478Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #5 to worker0, as the agent is shutting down 2022-12-01T10:40:42.8670550Z ok (11.158s) 2022-12-01T10:40:42.8670555Z 2022-12-01T10:40:42.8670756Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8670835Z Ran 1 test in 11.158s 2022-12-01T10:40:42.8670840Z 2022-12-01T10:40:42.8670901Z OK 2022-12-01T10:40:42.8670906Z 2022-12-01T10:40:42.8670995Z Generating XML reports... 2022-12-01T10:40:42.8671336Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102250.xml 2022-12-01T10:40:42.8671612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8671725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8672009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8672147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8672331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_wql2q2 2022-12-01T10:40:42.8672525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_wql2q2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8672781Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8672788Z 2022-12-01T10:40:42.8672863Z Running tests... 2022-12-01T10:40:42.8673065Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8673266Z test_async_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11074 2022-12-01T10:40:42.8673426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11075 2022-12-01T10:40:42.8673620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11076 2022-12-01T10:40:42.8673775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11077 2022-12-01T10:40:42.8674049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8674173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8674457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8674626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8674812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzo4nct1 2022-12-01T10:40:42.8674994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzo4nct1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8675269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8675394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8675673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8675809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8675992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnemz_bcy 2022-12-01T10:40:42.8676190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnemz_bcy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8676459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8676569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8676847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8676983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8677167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gfwgtt7 2022-12-01T10:40:42.8677362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gfwgtt7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8677632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8677757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8678036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8678171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8678500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcur6zk00 2022-12-01T10:40:42.8678696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcur6zk00/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8678861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8679028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8679191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8679353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8679425Z ok (2.141s) 2022-12-01T10:40:42.8679431Z 2022-12-01T10:40:42.8679630Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8679693Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8679698Z 2022-12-01T10:40:42.8679761Z OK 2022-12-01T10:40:42.8679766Z 2022-12-01T10:40:42.8679851Z Generating XML reports... 2022-12-01T10:40:42.8680274Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102303.xml 2022-12-01T10:40:42.8680554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8680680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8680960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8681095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8681320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvakme_x 2022-12-01T10:40:42.8681504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvakme_x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8681764Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8681769Z 2022-12-01T10:40:42.8681843Z Running tests... 2022-12-01T10:40:42.8682043Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8682268Z test_async_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11297 2022-12-01T10:40:42.8682429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11298 2022-12-01T10:40:42.8682585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11299 2022-12-01T10:40:42.8682740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11300 2022-12-01T10:40:42.8683000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8683130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8683411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8683546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8683735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40zmhn20 2022-12-01T10:40:42.8683930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40zmhn20/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8684201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8684327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8684608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8684729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8684915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblitm0m5 2022-12-01T10:40:42.8685109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblitm0m5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8685382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8685506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8685789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8685922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8686109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotw1fq4g 2022-12-01T10:40:42.8686289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotw1fq4g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8686557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8686680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8686988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8687124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8687307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2da3i_mm 2022-12-01T10:40:42.8687502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2da3i_mm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8687668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8687932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8688082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8688247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8688316Z ok (2.138s) 2022-12-01T10:40:42.8688321Z 2022-12-01T10:40:42.8688523Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8688602Z Ran 1 test in 2.138s 2022-12-01T10:40:42.8688607Z 2022-12-01T10:40:42.8688669Z OK 2022-12-01T10:40:42.8688674Z 2022-12-01T10:40:42.8688758Z Generating XML reports... 2022-12-01T10:40:42.8689102Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102306.xml 2022-12-01T10:40:42.8689360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8689487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8689767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8689902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8690084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4j_244b6 2022-12-01T10:40:42.8690278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4j_244b6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8690531Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8690536Z 2022-12-01T10:40:42.8690611Z Running tests... 2022-12-01T10:40:42.8690807Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8691023Z test_async_class_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11520 2022-12-01T10:40:42.8691183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11521 2022-12-01T10:40:42.8691338Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11522 2022-12-01T10:40:42.8691493Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11523 2022-12-01T10:40:42.8691768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8691890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8692171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8692307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8692493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrei7mek 2022-12-01T10:40:42.8692679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrei7mek/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8692948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8693072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8693350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8693530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8693716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvo4fzhj 2022-12-01T10:40:42.8693913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvo4fzhj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8694188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8694325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8694605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8694739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8694922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9l97e0k 2022-12-01T10:40:42.8695117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9l97e0k/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8695386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8695509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8695785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8695922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8696092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pf8anvh 2022-12-01T10:40:42.8696287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pf8anvh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8696452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8696618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8696780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8696943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8697012Z ok (2.040s) 2022-12-01T10:40:42.8697017Z 2022-12-01T10:40:42.8697215Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8697279Z Ran 1 test in 2.041s 2022-12-01T10:40:42.8697286Z 2022-12-01T10:40:42.8697347Z OK 2022-12-01T10:40:42.8697351Z 2022-12-01T10:40:42.8697435Z Generating XML reports... 2022-12-01T10:40:42.8697774Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102310.xml 2022-12-01T10:40:42.8698046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8698172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8698458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8698595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8698783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp_vejy4 2022-12-01T10:40:42.8698963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp_vejy4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8699223Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8699228Z 2022-12-01T10:40:42.8699304Z Running tests... 2022-12-01T10:40:42.8699501Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8699729Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11743 2022-12-01T10:40:42.8699922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11744 2022-12-01T10:40:42.8700082Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11745 2022-12-01T10:40:42.8700238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11746 2022-12-01T10:40:42.8700501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8700628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8700961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8701099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8701286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02xutgbu 2022-12-01T10:40:42.8701483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02xutgbu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8701759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8701886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8702168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8702292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8702482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplk39hvez 2022-12-01T10:40:42.8702676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplk39hvez/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8702948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8703073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8703355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8703491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8703676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw9cd1o2 2022-12-01T10:40:42.8703858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw9cd1o2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8704136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8704261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8704543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8704680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8704867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphx6am6xy 2022-12-01T10:40:42.8705064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphx6am6xy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8705230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8705398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8705546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8705711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8705780Z ok (2.141s) 2022-12-01T10:40:42.8705785Z 2022-12-01T10:40:42.8705986Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8706067Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8706072Z 2022-12-01T10:40:42.8706134Z OK 2022-12-01T10:40:42.8706139Z 2022-12-01T10:40:42.8706226Z Generating XML reports... 2022-12-01T10:40:42.8706597Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102313.xml 2022-12-01T10:40:42.8706858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8706984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8707265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8707432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8707620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprca24dnd 2022-12-01T10:40:42.8707818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprca24dnd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8708077Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8708085Z 2022-12-01T10:40:42.8708161Z Running tests... 2022-12-01T10:40:42.8708360Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8708578Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11966 2022-12-01T10:40:42.8708737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11967 2022-12-01T10:40:42.8708897Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11968 2022-12-01T10:40:42.8709054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11969 2022-12-01T10:40:42.8709328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8709455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8709740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8709877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8710063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptn8oxfb3 2022-12-01T10:40:42.8710246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptn8oxfb3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8710518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8710645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8710924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8711062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8711247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5k21wwwp 2022-12-01T10:40:42.8711445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5k21wwwp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8711718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8711829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8712109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8712248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8712433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmca4kjx 2022-12-01T10:40:42.8712632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmca4kjx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8712903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8713062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8713345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8713480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8713651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2x0mopcm 2022-12-01T10:40:42.8713847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2x0mopcm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8714043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8714208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8714373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8714538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8714611Z ok (2.242s) 2022-12-01T10:40:42.8714616Z 2022-12-01T10:40:42.8714817Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8714881Z Ran 1 test in 2.242s 2022-12-01T10:40:42.8714886Z 2022-12-01T10:40:42.8714948Z OK 2022-12-01T10:40:42.8714953Z 2022-12-01T10:40:42.8715039Z Generating XML reports... 2022-12-01T10:40:42.8715379Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102317.xml 2022-12-01T10:40:42.8715654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8715781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8716062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8716200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8716391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqerliv72 2022-12-01T10:40:42.8716574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqerliv72/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8716832Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8716837Z 2022-12-01T10:40:42.8716913Z Running tests... 2022-12-01T10:40:42.8717112Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8717349Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12189 2022-12-01T10:40:42.8717508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12190 2022-12-01T10:40:42.8717665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12191 2022-12-01T10:40:42.8717825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12192 2022-12-01T10:40:42.8718088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8718346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8718633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8718770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8718958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4_l5r26 2022-12-01T10:40:42.8719154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4_l5r26/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8719425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8719551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8719910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8720033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8720218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcmngce8 2022-12-01T10:40:42.8720416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcmngce8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8720756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8720880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8721160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8721296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8721483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vgd8i_t 2022-12-01T10:40:42.8721665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vgd8i_t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8721937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8722059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8722340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8722475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8722659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jj328tj 2022-12-01T10:40:42.8722852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jj328tj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8723021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8723187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8723337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8723503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8723574Z ok (2.139s) 2022-12-01T10:40:42.8723582Z 2022-12-01T10:40:42.8723779Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8723857Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8723862Z 2022-12-01T10:40:42.8723924Z OK 2022-12-01T10:40:42.8723928Z 2022-12-01T10:40:42.8724015Z Generating XML reports... 2022-12-01T10:40:42.8724358Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102321.xml 2022-12-01T10:40:42.8724632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8724745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8725023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8725161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8725349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd98z74n_ 2022-12-01T10:40:42.8725546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd98z74n_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8725803Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8725807Z 2022-12-01T10:40:42.8725883Z Running tests... 2022-12-01T10:40:42.8726082Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8726330Z test_async_function_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12412 2022-12-01T10:40:42.8726494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12413 2022-12-01T10:40:42.8726650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12414 2022-12-01T10:40:42.8726806Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12415 2022-12-01T10:40:42.8727080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8727234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8727519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8727654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8727902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yg2xk0u 2022-12-01T10:40:42.8728087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yg2xk0u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8728368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8728491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8728774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8728909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8729093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3fe7o1z 2022-12-01T10:40:42.8729288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3fe7o1z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8729561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8729671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8729952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8730088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8730273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3ol5r4o 2022-12-01T10:40:42.8730470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3ol5r4o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8730740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8730864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8731145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8731283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8731450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_mdw8ay 2022-12-01T10:40:42.8731644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_mdw8ay/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8731807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8731971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8732137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8732298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8732366Z ok (2.142s) 2022-12-01T10:40:42.8732371Z 2022-12-01T10:40:42.8732570Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8732633Z Ran 1 test in 2.142s 2022-12-01T10:40:42.8732689Z 2022-12-01T10:40:42.8732738Z OK 2022-12-01T10:40:42.8732743Z 2022-12-01T10:40:42.8732830Z Generating XML reports... 2022-12-01T10:40:42.8733170Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102324.xml 2022-12-01T10:40:42.8733445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8733568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8733877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8734013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8734203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73u23a8t 2022-12-01T10:40:42.8734386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73u23a8t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8734645Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8734651Z 2022-12-01T10:40:42.8734725Z Running tests... 2022-12-01T10:40:42.8734924Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8735158Z test_async_function_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12635 2022-12-01T10:40:42.8735320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12636 2022-12-01T10:40:42.8735476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12637 2022-12-01T10:40:42.8735631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12638 2022-12-01T10:40:42.8735889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8736016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8736295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8736432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8736617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4krin7p3 2022-12-01T10:40:42.8736814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4krin7p3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8737088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8737213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8737491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8737611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8737797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3eelt40v 2022-12-01T10:40:42.8737995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3eelt40v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8738266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8738388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8738667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8738801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8738984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wfd782g 2022-12-01T10:40:42.8739254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8739407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8739688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8739824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8740011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9sq53pl 2022-12-01T10:40:42.8740232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wfd782g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8740424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9sq53pl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8740590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8740756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8740907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8741070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8741140Z ok (2.043s) 2022-12-01T10:40:42.8741145Z 2022-12-01T10:40:42.8741344Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8741420Z Ran 1 test in 2.043s 2022-12-01T10:40:42.8741425Z 2022-12-01T10:40:42.8741485Z OK 2022-12-01T10:40:42.8741490Z 2022-12-01T10:40:42.8741577Z Generating XML reports... 2022-12-01T10:40:42.8741920Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102328.xml 2022-12-01T10:40:42.8742192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8742303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8742588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8742724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8742912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ufsbv_p 2022-12-01T10:40:42.8743111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ufsbv_p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8743369Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8743376Z 2022-12-01T10:40:42.8743452Z Running tests... 2022-12-01T10:40:42.8743648Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8743868Z test_async_function_multi_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12858 2022-12-01T10:40:42.8744026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12859 2022-12-01T10:40:42.8744182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12860 2022-12-01T10:40:42.8744335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12861 2022-12-01T10:40:42.8744613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8744738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8745018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8745156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8745342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6y8ewe3 2022-12-01T10:40:42.8745523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6y8ewe3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8745826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8745951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8746229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8746364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8746549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8u929prg 2022-12-01T10:40:42.8746770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8u929prg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8747040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8747150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8747429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8747565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8747748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fxid30b 2022-12-01T10:40:42.8747942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fxid30b/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8748211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8748335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8748614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8748751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8748922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposxinyy0 2022-12-01T10:40:42.8749122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposxinyy0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8749286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8749450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8749619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8749780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8749852Z ok (2.039s) 2022-12-01T10:40:42.8749857Z 2022-12-01T10:40:42.8750058Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8750121Z Ran 1 test in 2.039s 2022-12-01T10:40:42.8750140Z 2022-12-01T10:40:42.8750189Z OK 2022-12-01T10:40:42.8750194Z 2022-12-01T10:40:42.8750279Z Generating XML reports... 2022-12-01T10:40:42.8750631Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102331.xml 2022-12-01T10:40:42.8750907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8751019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8751296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8751429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8751615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcsohbtt 2022-12-01T10:40:42.8751810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcsohbtt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8752062Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8752067Z 2022-12-01T10:40:42.8752141Z Running tests... 2022-12-01T10:40:42.8752371Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8752612Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13081 2022-12-01T10:40:42.8752756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13082 2022-12-01T10:40:42.8752910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13083 2022-12-01T10:40:42.8753092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13084 2022-12-01T10:40:42.8753365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8753487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8753767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8753902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8754086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_cm_xku 2022-12-01T10:40:42.8754265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_cm_xku/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8754535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8754661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8754938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8755072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8755256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2kn0es6 2022-12-01T10:40:42.8755452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2kn0es6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8755723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8755847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8756112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8756248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8756435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiakwp035 2022-12-01T10:40:42.8756629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiakwp035/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8756900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8757023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8757306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8757446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8757615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwulcuhd 2022-12-01T10:40:42.8757811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwulcuhd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8757979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8758144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8758437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8758601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8758672Z ok (2.138s) 2022-12-01T10:40:42.8758678Z 2022-12-01T10:40:42.8758928Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8759005Z Ran 1 test in 2.138s 2022-12-01T10:40:42.8759011Z 2022-12-01T10:40:42.8759058Z OK 2022-12-01T10:40:42.8759062Z 2022-12-01T10:40:42.8759149Z Generating XML reports... 2022-12-01T10:40:42.8759491Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102335.xml 2022-12-01T10:40:42.8759802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8759926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8760205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8760341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8760530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj24ws_m8 2022-12-01T10:40:42.8760710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj24ws_m8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8760968Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8760973Z 2022-12-01T10:40:42.8761048Z Running tests... 2022-12-01T10:40:42.8761247Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8761490Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13304 2022-12-01T10:40:42.8761648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13305 2022-12-01T10:40:42.8761804Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13306 2022-12-01T10:40:42.8761960Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13307 2022-12-01T10:40:42.8762238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8762351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8762631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8762764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8762952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphg_w4bgq 2022-12-01T10:40:42.8763147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphg_w4bgq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8763421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8763543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8763825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8763962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8764132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfod38a9g 2022-12-01T10:40:42.8764328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfod38a9g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8764595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8764721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8764997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8765131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8765312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3icmy4yj 2022-12-01T10:40:42.8765531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3icmy4yj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8765792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8765913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8766191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8766352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8766534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5sbnynw 2022-12-01T10:40:42.8766725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5sbnynw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8766890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8767056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8767217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8767365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8767432Z ok (2.139s) 2022-12-01T10:40:42.8767437Z 2022-12-01T10:40:42.8767632Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8767761Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8767767Z 2022-12-01T10:40:42.8767831Z OK 2022-12-01T10:40:42.8767835Z 2022-12-01T10:40:42.8767921Z Generating XML reports... 2022-12-01T10:40:42.8768262Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102338.xml 2022-12-01T10:40:42.8768533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8768647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8768925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8769059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8769243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfho61lch 2022-12-01T10:40:42.8769437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfho61lch/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8769693Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8769699Z 2022-12-01T10:40:42.8769774Z Running tests... 2022-12-01T10:40:42.8769969Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8770202Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13527 2022-12-01T10:40:42.8770348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13528 2022-12-01T10:40:42.8770505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13529 2022-12-01T10:40:42.8770659Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13530 2022-12-01T10:40:42.8770932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8771055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8771335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8771469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8771654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwy51w8j 2022-12-01T10:40:42.8771879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwy51w8j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8772151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8772274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8772550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8772710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8772893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7t3kvwd 2022-12-01T10:40:42.8773085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7t3kvwd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8773355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8773480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8773744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8773877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8774063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbpmnuo_ 2022-12-01T10:40:42.8774258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbpmnuo_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8774531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8774654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8774933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8775067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8775253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip97qry_ 2022-12-01T10:40:42.8775434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip97qry_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8775596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8775759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8775924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8776086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8776154Z ok (2.038s) 2022-12-01T10:40:42.8776159Z 2022-12-01T10:40:42.8776354Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8776428Z Ran 1 test in 2.039s 2022-12-01T10:40:42.8776433Z 2022-12-01T10:40:42.8776480Z OK 2022-12-01T10:40:42.8776484Z 2022-12-01T10:40:42.8776569Z Generating XML reports... 2022-12-01T10:40:42.8776906Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102342.xml 2022-12-01T10:40:42.8777179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8777301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8777583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8777720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8777904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrg1p038 2022-12-01T10:40:42.8778084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrg1p038/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8778372Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8778378Z 2022-12-01T10:40:42.8778450Z Running tests... 2022-12-01T10:40:42.8778646Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8778885Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13750 2022-12-01T10:40:42.8779044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13751 2022-12-01T10:40:42.8779236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13752 2022-12-01T10:40:42.8779390Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13753 2022-12-01T10:40:42.8779660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8779770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8780051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8780187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8780371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mge0lzy 2022-12-01T10:40:42.8780566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mge0lzy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8780840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8780963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8781243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8781377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8781549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1ksyjlu 2022-12-01T10:40:42.8781820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8781941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8782218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8782351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8782536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgvqfzop 2022-12-01T10:40:42.8782730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1ksyjlu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8782925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgvqfzop/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8783186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8783307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8783583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8783717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8783898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3su97bl 2022-12-01T10:40:42.8784092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3su97bl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8784256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8784418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8784578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8784780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8784848Z ok (2.139s) 2022-12-01T10:40:42.8784853Z 2022-12-01T10:40:42.8785050Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8785125Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8785130Z 2022-12-01T10:40:42.8785189Z OK 2022-12-01T10:40:42.8785193Z 2022-12-01T10:40:42.8785276Z Generating XML reports... 2022-12-01T10:40:42.8785643Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102346.xml 2022-12-01T10:40:42.8785913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8786023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8786301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8786436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8786617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetc9_e14 2022-12-01T10:40:42.8786810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetc9_e14/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8787067Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8787074Z 2022-12-01T10:40:42.8787149Z Running tests... 2022-12-01T10:40:42.8787349Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8787591Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13973 2022-12-01T10:40:42.8787735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13974 2022-12-01T10:40:42.8787889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13975 2022-12-01T10:40:42.8788046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13976 2022-12-01T10:40:42.8788318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8788438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8788719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8788855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8789036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpid70k297 2022-12-01T10:40:42.8789215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpid70k297/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8789484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8789608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8789886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8790020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8790202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7eyic38 2022-12-01T10:40:42.8790395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7eyic38/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8790667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8790791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8791058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8791220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8791403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmhz8xbl 2022-12-01T10:40:42.8791597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmhz8xbl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8791868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8791989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8792296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8792429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8792610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpjj_s9z 2022-12-01T10:40:42.8792791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpjj_s9z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8792955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8793118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8793278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8793438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8793508Z ok (2.139s) 2022-12-01T10:40:42.8793513Z 2022-12-01T10:40:42.8793712Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8793788Z Ran 1 test in 2.140s 2022-12-01T10:40:42.8793793Z 2022-12-01T10:40:42.8793841Z OK 2022-12-01T10:40:42.8793845Z 2022-12-01T10:40:42.8793927Z Generating XML reports... 2022-12-01T10:40:42.8794265Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102349.xml 2022-12-01T10:40:42.8794538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8794661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8794937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8795073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8795258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_uiv52s 2022-12-01T10:40:42.8795449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_uiv52s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8795689Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8795694Z 2022-12-01T10:40:42.8795767Z Running tests... 2022-12-01T10:40:42.8795962Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8796189Z test_async_function_nested (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14196 2022-12-01T10:40:42.8796345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14197 2022-12-01T10:40:42.8796500Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14198 2022-12-01T10:40:42.8796651Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14199 2022-12-01T10:40:42.8796926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8797036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8797317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8797452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8797672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe11rq9ag 2022-12-01T10:40:42.8797868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe11rq9ag/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8798140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8798409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8798753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8798891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8799063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nrvw5y5 2022-12-01T10:40:42.8799259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nrvw5y5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8799533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8799660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8799940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8800076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8800262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf6_4jej 2022-12-01T10:40:42.8800458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf6_4jej/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8800717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8800840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8801120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8801258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8801441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1nlxbi8x 2022-12-01T10:40:42.8801636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1nlxbi8x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8801799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8801964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8802125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8802273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8802340Z ok (2.241s) 2022-12-01T10:40:42.8802345Z 2022-12-01T10:40:42.8802545Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8802622Z Ran 1 test in 2.242s 2022-12-01T10:40:42.8802627Z 2022-12-01T10:40:42.8802686Z OK 2022-12-01T10:40:42.8802691Z 2022-12-01T10:40:42.8802774Z Generating XML reports... 2022-12-01T10:40:42.8803114Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102353.xml 2022-12-01T10:40:42.8803388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8803501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8803783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8803917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8804104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps34sg_ds 2022-12-01T10:40:42.8804339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps34sg_ds/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8804595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8804600Z 2022-12-01T10:40:42.8804673Z Running tests... 2022-12-01T10:40:42.8804871Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8805105Z test_async_function_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14419 2022-12-01T10:40:42.8805283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14420 2022-12-01T10:40:42.8805438Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14421 2022-12-01T10:40:42.8805594Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14422 2022-12-01T10:40:42.8805867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8805991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8806270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8806404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8806589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7e49w7nb 2022-12-01T10:40:42.8806773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7e49w7nb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8807044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8807166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8807445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8807581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8807828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbr09c1j 2022-12-01T10:40:42.8808025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbr09c1j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8808301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8808428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8808692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8808830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8809018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwocy42l 2022-12-01T10:40:42.8809214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwocy42l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8809484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8809611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8809891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8810027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8810215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifd3hwcx 2022-12-01T10:40:42.8810398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifd3hwcx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8810563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8810728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8810935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8811098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8811168Z ok (2.140s) 2022-12-01T10:40:42.8811173Z 2022-12-01T10:40:42.8811372Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8811449Z Ran 1 test in 2.140s 2022-12-01T10:40:42.8811484Z 2022-12-01T10:40:42.8811532Z OK 2022-12-01T10:40:42.8811537Z 2022-12-01T10:40:42.8811626Z Generating XML reports... 2022-12-01T10:40:42.8811967Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102356.xml 2022-12-01T10:40:42.8812239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8812366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8812647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8812784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8812969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztj7tatn 2022-12-01T10:40:42.8813167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztj7tatn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8813411Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8813416Z 2022-12-01T10:40:42.8813493Z Running tests... 2022-12-01T10:40:42.8813693Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8813918Z test_async_function_raise (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14642 2022-12-01T10:40:42.8814082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14643 2022-12-01T10:40:42.8814238Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14644 2022-12-01T10:40:42.8814392Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14645 2022-12-01T10:40:42.8814667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8814778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8815064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8815201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8815386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_51i_yyj 2022-12-01T10:40:42.8815580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_51i_yyj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8815854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8815978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8816262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8816398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8816573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv5potg0 2022-12-01T10:40:42.8816847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8816971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8817250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8817423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8817608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtf2j0yv 2022-12-01T10:40:42.8817803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv5potg0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8817993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtf2j0yv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8818251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8818408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8818687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8818822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8819008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpce3ce01h 2022-12-01T10:40:42.8819204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpce3ce01h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8819370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8819532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8819694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8819844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8819935Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.8820058Z RuntimeError('Expected error') 2022-12-01T10:40:42.8820151Z Traceback (most recent call last): 2022-12-01T10:40:42.8820411Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8820549Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8820807Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-12-01T10:40:42.8820888Z return fn(*args, **kwargs) 2022-12-01T10:40:42.8821166Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-12-01T10:40:42.8821267Z raise RuntimeError("Expected error") 2022-12-01T10:40:42.8821357Z RuntimeError: Expected error 2022-12-01T10:40:42.8821365Z 2022-12-01T10:40:42.8821455Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.8821574Z RuntimeError('Expected error') 2022-12-01T10:40:42.8821666Z Traceback (most recent call last): 2022-12-01T10:40:42.8821924Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8822044Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8822297Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-12-01T10:40:42.8822380Z return fn(*args, **kwargs) 2022-12-01T10:40:42.8822667Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-12-01T10:40:42.8822767Z raise RuntimeError("Expected error") 2022-12-01T10:40:42.8822854Z RuntimeError: Expected error 2022-12-01T10:40:42.8822859Z 2022-12-01T10:40:42.8822950Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.8823072Z RuntimeError('Expected error') 2022-12-01T10:40:42.8823150Z Traceback (most recent call last): 2022-12-01T10:40:42.8823409Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8823540Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8823787Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-12-01T10:40:42.8823926Z return fn(*args, **kwargs) 2022-12-01T10:40:42.8824219Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-12-01T10:40:42.8824322Z raise RuntimeError("Expected error") 2022-12-01T10:40:42.8824410Z RuntimeError: Expected error 2022-12-01T10:40:42.8824415Z 2022-12-01T10:40:42.8824492Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.8824613Z RuntimeError('Expected error') 2022-12-01T10:40:42.8824730Z Traceback (most recent call last): 2022-12-01T10:40:42.8824984Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8825116Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8825365Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-12-01T10:40:42.8825449Z return fn(*args, **kwargs) 2022-12-01T10:40:42.8825735Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-12-01T10:40:42.8825823Z raise RuntimeError("Expected error") 2022-12-01T10:40:42.8825913Z RuntimeError: Expected error 2022-12-01T10:40:42.8825918Z 2022-12-01T10:40:42.8825986Z ok (2.140s) 2022-12-01T10:40:42.8825990Z 2022-12-01T10:40:42.8826190Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8826266Z Ran 1 test in 2.140s 2022-12-01T10:40:42.8826273Z 2022-12-01T10:40:42.8826335Z OK 2022-12-01T10:40:42.8826340Z 2022-12-01T10:40:42.8826425Z Generating XML reports... 2022-12-01T10:40:42.8826770Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102400.xml 2022-12-01T10:40:42.8827032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8827155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8827439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8827576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8827762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9ze5dh2 2022-12-01T10:40:42.8827961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9ze5dh2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8828221Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8828226Z 2022-12-01T10:40:42.8828299Z Running tests... 2022-12-01T10:40:42.8828497Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8829182Z test_async_function_raise_async (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76907 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.218s) 2022-12-01T10:40:42.8829201Z 2022-12-01T10:40:42.8829383Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8829457Z Ran 1 test in 0.218s 2022-12-01T10:40:42.8829461Z 2022-12-01T10:40:42.8829532Z OK (skipped=1) 2022-12-01T10:40:42.8829537Z 2022-12-01T10:40:42.8829623Z Generating XML reports... 2022-12-01T10:40:42.8829960Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102403.xml 2022-12-01T10:40:42.8830231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8830358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8830711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8830847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8831021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperwhq5dp 2022-12-01T10:40:42.8831220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperwhq5dp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8831475Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8831510Z 2022-12-01T10:40:42.8831583Z Running tests... 2022-12-01T10:40:42.8831781Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8832013Z test_async_function_raise_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14875 2022-12-01T10:40:42.8832172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14876 2022-12-01T10:40:42.8832329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14877 2022-12-01T10:40:42.8832470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14878 2022-12-01T10:40:42.8832746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8832872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8833152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8833292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8833479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4d20my_d 2022-12-01T10:40:42.8833678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4d20my_d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8833953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8834075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8834342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8834474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8834654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1b_iskbc 2022-12-01T10:40:42.8834847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1b_iskbc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8835117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8835239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8835518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8835654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8835837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpropyt5gy 2022-12-01T10:40:42.8836019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpropyt5gy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8836287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8836411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8836689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8836822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8837003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lukb8dh 2022-12-01T10:40:42.8837233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lukb8dh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8837398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8837548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8837710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8837871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8837990Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.8838115Z RuntimeError('Expected error') 2022-12-01T10:40:42.8838364Z Traceback (most recent call last): 2022-12-01T10:40:42.8838631Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8838767Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8839015Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-12-01T10:40:42.8839100Z return fn(*args, **kwargs) 2022-12-01T10:40:42.8839392Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-12-01T10:40:42.8839495Z raise RuntimeError("Expected error") 2022-12-01T10:40:42.8839587Z RuntimeError: Expected error 2022-12-01T10:40:42.8839592Z 2022-12-01T10:40:42.8839687Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.8839810Z RuntimeError('Expected error') 2022-12-01T10:40:42.8839902Z Traceback (most recent call last): 2022-12-01T10:40:42.8840147Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8840285Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8840537Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-12-01T10:40:42.8840623Z return fn(*args, **kwargs) 2022-12-01T10:40:42.8840914Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-12-01T10:40:42.8841014Z raise RuntimeError("Expected error") 2022-12-01T10:40:42.8841103Z RuntimeError: Expected error 2022-12-01T10:40:42.8841109Z 2022-12-01T10:40:42.8841198Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.8841307Z RuntimeError('Expected error') 2022-12-01T10:40:42.8841398Z Traceback (most recent call last): 2022-12-01T10:40:42.8841654Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8841783Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8842033Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-12-01T10:40:42.8842113Z return fn(*args, **kwargs) 2022-12-01T10:40:42.8842403Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-12-01T10:40:42.8842503Z raise RuntimeError("Expected error") 2022-12-01T10:40:42.8842579Z RuntimeError: Expected error 2022-12-01T10:40:42.8842584Z 2022-12-01T10:40:42.8842675Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.8842797Z RuntimeError('Expected error') 2022-12-01T10:40:42.8842887Z Traceback (most recent call last): 2022-12-01T10:40:42.8843142Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.8843273Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.8843522Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-12-01T10:40:42.8843589Z return fn(*args, **kwargs) 2022-12-01T10:40:42.8843933Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-12-01T10:40:42.8844039Z raise RuntimeError("Expected error") 2022-12-01T10:40:42.8844132Z RuntimeError: Expected error 2022-12-01T10:40:42.8844137Z 2022-12-01T10:40:42.8844204Z ok (2.050s) 2022-12-01T10:40:42.8844209Z 2022-12-01T10:40:42.8844409Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8844486Z Ran 1 test in 2.050s 2022-12-01T10:40:42.8844491Z 2022-12-01T10:40:42.8844588Z OK 2022-12-01T10:40:42.8844592Z 2022-12-01T10:40:42.8844665Z Generating XML reports... 2022-12-01T10:40:42.8845007Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102405.xml 2022-12-01T10:40:42.8849862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8850031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8850343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8850486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8850678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7k0uqqx 2022-12-01T10:40:42.8850881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7k0uqqx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8851142Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8851153Z 2022-12-01T10:40:42.8851229Z Running tests... 2022-12-01T10:40:42.8851429Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8851660Z test_async_function_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15098 2022-12-01T10:40:42.8851818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15099 2022-12-01T10:40:42.8851963Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15100 2022-12-01T10:40:42.8852118Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15101 2022-12-01T10:40:42.8852395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8852521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8852809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8852950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8853138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht0govgt 2022-12-01T10:40:42.8853337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht0govgt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8853599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8853725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8854006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8854145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8854333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo17s08ol 2022-12-01T10:40:42.8854529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo17s08ol/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8854800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8854926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8855295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8855420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8855604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tt86w3v 2022-12-01T10:40:42.8855799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tt86w3v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8856071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8856234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8856515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8856651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8856838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ys0ndmq 2022-12-01T10:40:42.8857036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ys0ndmq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8857188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8857355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8857520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8857686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8857755Z ok (2.141s) 2022-12-01T10:40:42.8857760Z 2022-12-01T10:40:42.8857958Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8858035Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8858040Z 2022-12-01T10:40:42.8858102Z OK 2022-12-01T10:40:42.8858108Z 2022-12-01T10:40:42.8858180Z Generating XML reports... 2022-12-01T10:40:42.8858530Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102409.xml 2022-12-01T10:40:42.8858803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8858929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8859212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8859351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8859539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg810oizz 2022-12-01T10:40:42.8859735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg810oizz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8859992Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8859997Z 2022-12-01T10:40:42.8860058Z Running tests... 2022-12-01T10:40:42.8860257Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8860494Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15321 2022-12-01T10:40:42.8860652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15322 2022-12-01T10:40:42.8860809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15323 2022-12-01T10:40:42.8860968Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15324 2022-12-01T10:40:42.8861242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8861366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8861635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8861817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8862007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbay8pkuq 2022-12-01T10:40:42.8862205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbay8pkuq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8862478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8862634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8862915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8863050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8863235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbt21s2c 2022-12-01T10:40:42.8863420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbt21s2c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8863690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8863813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8864090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8864225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8864412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62mtbxhv 2022-12-01T10:40:42.8864607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62mtbxhv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8864878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8864988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8865270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8865405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8865591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0q3enzgb 2022-12-01T10:40:42.8865788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0q3enzgb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8865955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8866119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8866282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8866445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8866501Z ok (2.153s) 2022-12-01T10:40:42.8866510Z 2022-12-01T10:40:42.8866709Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8866784Z Ran 1 test in 2.153s 2022-12-01T10:40:42.8866789Z 2022-12-01T10:40:42.8866851Z OK 2022-12-01T10:40:42.8866855Z 2022-12-01T10:40:42.8866940Z Generating XML reports... 2022-12-01T10:40:42.8867281Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102412.xml 2022-12-01T10:40:42.8867557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8867684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8867950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8868086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8868310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt4f3wkh 2022-12-01T10:40:42.8868508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt4f3wkh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8868764Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8868769Z 2022-12-01T10:40:42.8868844Z Running tests... 2022-12-01T10:40:42.8869041Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8869323Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15544 2022-12-01T10:40:42.8869482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15545 2022-12-01T10:40:42.8869626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15546 2022-12-01T10:40:42.8869781Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15547 2022-12-01T10:40:42.8870061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8870189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8870470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8870606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8870801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei23mpxw 2022-12-01T10:40:42.8871064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei23mpxw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8871337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8871449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8871732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8871867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8872053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpscoflhzv 2022-12-01T10:40:42.8872251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpscoflhzv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8872522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8872650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8872929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8873051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8873237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_gud1o_ 2022-12-01T10:40:42.8873432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_gud1o_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8873704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8873829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8874106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8874244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8874429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrrfrk1y 2022-12-01T10:40:42.8874625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrrfrk1y/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8874777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8874978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8875143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8875305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8875374Z ok (2.038s) 2022-12-01T10:40:42.8875380Z 2022-12-01T10:40:42.8875580Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8875687Z Ran 1 test in 2.039s 2022-12-01T10:40:42.8875692Z 2022-12-01T10:40:42.8875753Z OK 2022-12-01T10:40:42.8875758Z 2022-12-01T10:40:42.8875829Z Generating XML reports... 2022-12-01T10:40:42.8876167Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102416.xml 2022-12-01T10:40:42.8876439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8876567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8876846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8876981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8877166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpys6_r_ws 2022-12-01T10:40:42.8877364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpys6_r_ws/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8877619Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8877625Z 2022-12-01T10:40:42.8877686Z Running tests... 2022-12-01T10:40:42.8877883Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8878122Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15767 2022-12-01T10:40:42.8878534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15768 2022-12-01T10:40:42.8878690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15769 2022-12-01T10:40:42.8878845Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15770 2022-12-01T10:40:42.8879125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8879253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8879522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8879659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8879845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0a1q1kvz 2022-12-01T10:40:42.8880042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0a1q1kvz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8880317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8880441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8880721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8880860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8881045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl047yzr 2022-12-01T10:40:42.8881225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl047yzr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8881497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8881717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8882000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8882135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8882321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lw46phb 2022-12-01T10:40:42.8882515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lw46phb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8882828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8882939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8883221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8883359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8883542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgado7jd4 2022-12-01T10:40:42.8883737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgado7jd4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8883901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8884069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8884235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8884399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8884455Z ok (2.037s) 2022-12-01T10:40:42.8884460Z 2022-12-01T10:40:42.8884661Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8884739Z Ran 1 test in 2.038s 2022-12-01T10:40:42.8884744Z 2022-12-01T10:40:42.8884807Z OK 2022-12-01T10:40:42.8884815Z 2022-12-01T10:40:42.8884900Z Generating XML reports... 2022-12-01T10:40:42.8885240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102419.xml 2022-12-01T10:40:42.8885514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8885637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8885904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8886039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8886225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykb04py4 2022-12-01T10:40:42.8886420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykb04py4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8886678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8886683Z 2022-12-01T10:40:42.8886757Z Running tests... 2022-12-01T10:40:42.8886955Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8887199Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15990 2022-12-01T10:40:42.8887361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15991 2022-12-01T10:40:42.8887504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15992 2022-12-01T10:40:42.8887658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15993 2022-12-01T10:40:42.8888021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8888147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8888473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8888612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8888801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsx3zn5u 2022-12-01T10:40:42.8889000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsx3zn5u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8889305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8889415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8889696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8889833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8890022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8hwz3ak 2022-12-01T10:40:42.8890218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8hwz3ak/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8890490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8890613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8890896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8891019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8891209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mksh3s1 2022-12-01T10:40:42.8891403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mksh3s1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8891677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8891801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8892082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8892219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8892403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60i9gsc8 2022-12-01T10:40:42.8892598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60i9gsc8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8892749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8892913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8893078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8893244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8893313Z ok (2.040s) 2022-12-01T10:40:42.8893318Z 2022-12-01T10:40:42.8893520Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8893596Z Ran 1 test in 2.040s 2022-12-01T10:40:42.8893601Z 2022-12-01T10:40:42.8893666Z OK 2022-12-01T10:40:42.8893670Z 2022-12-01T10:40:42.8893742Z Generating XML reports... 2022-12-01T10:40:42.8894084Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102423.xml 2022-12-01T10:40:42.8894357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8894482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8894805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8894944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8895132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptia0plsx 2022-12-01T10:40:42.8895330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptia0plsx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8895590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8895627Z 2022-12-01T10:40:42.8895687Z Running tests... 2022-12-01T10:40:42.8895887Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8896135Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16213 2022-12-01T10:40:42.8896294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16214 2022-12-01T10:40:42.8896452Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16215 2022-12-01T10:40:42.8896616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16216 2022-12-01T10:40:42.8896893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8897020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8897286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8897424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8897610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33ugde6_ 2022-12-01T10:40:42.8897809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33ugde6_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8898082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8898207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8898486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8898622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8898808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqx2b6zc 2022-12-01T10:40:42.8898992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqx2b6zc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8899268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8899391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8899669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8899804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8899989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpco5789o4 2022-12-01T10:40:42.8900180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpco5789o4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8900451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8900576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8900842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8900977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8901160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t4db5if 2022-12-01T10:40:42.8901390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t4db5if/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8901559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8901724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8901886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8902049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8902132Z ok (2.039s) 2022-12-01T10:40:42.8902137Z 2022-12-01T10:40:42.8902336Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8902412Z Ran 1 test in 2.039s 2022-12-01T10:40:42.8902417Z 2022-12-01T10:40:42.8902479Z OK 2022-12-01T10:40:42.8902483Z 2022-12-01T10:40:42.8902568Z Generating XML reports... 2022-12-01T10:40:42.8902910Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102426.xml 2022-12-01T10:40:42.8903186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8903310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8903589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8903713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8903900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3sl3_b9 2022-12-01T10:40:42.8904093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3sl3_b9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8904349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8904354Z 2022-12-01T10:40:42.8904427Z Running tests... 2022-12-01T10:40:42.8904629Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8904871Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16436 2022-12-01T10:40:42.8905029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16437 2022-12-01T10:40:42.8905171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16438 2022-12-01T10:40:42.8905329Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16439 2022-12-01T10:40:42.8905600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8905723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8906003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8906140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8906324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebk7_ss9 2022-12-01T10:40:42.8906521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebk7_ss9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8906795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8906906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8907190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8907324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8907509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bopztsb 2022-12-01T10:40:42.8907704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bopztsb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8908009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8908134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8908411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8908533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8908745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp208uttym 2022-12-01T10:40:42.8908939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp208uttym/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8909211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8909333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8909612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8909747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8909934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcxic4zh 2022-12-01T10:40:42.8910128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcxic4zh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8910282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8910448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8910611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8910776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8910844Z ok (2.139s) 2022-12-01T10:40:42.8910849Z 2022-12-01T10:40:42.8911049Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8911128Z Ran 1 test in 2.139s 2022-12-01T10:40:42.8911133Z 2022-12-01T10:40:42.8911194Z OK 2022-12-01T10:40:42.8911199Z 2022-12-01T10:40:42.8911270Z Generating XML reports... 2022-12-01T10:40:42.8911611Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102430.xml 2022-12-01T10:40:42.8911883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8912011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8912290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8912426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8912614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexrwwpoe 2022-12-01T10:40:42.8912813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexrwwpoe/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8913066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8913072Z 2022-12-01T10:40:42.8913133Z Running tests... 2022-12-01T10:40:42.8913331Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8913582Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16659 2022-12-01T10:40:42.8913744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16660 2022-12-01T10:40:42.8913899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16661 2022-12-01T10:40:42.8914053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16662 2022-12-01T10:40:42.8914355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8914482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8914764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8914885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8915134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuff_8837 2022-12-01T10:40:42.8915430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuff_8837/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8915709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8915832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8916116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8916252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8916435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlo_pogv 2022-12-01T10:40:42.8916614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlo_pogv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8916887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8917014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8917299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8917434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8917620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptux2czqr 2022-12-01T10:40:42.8917821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptux2czqr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8918092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8918353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8918622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8918761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8918943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp977ms1bn 2022-12-01T10:40:42.8919136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp977ms1bn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8919301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8919469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8919634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8919800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8919856Z ok (3.141s) 2022-12-01T10:40:42.8919862Z 2022-12-01T10:40:42.8920062Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8920140Z Ran 1 test in 3.141s 2022-12-01T10:40:42.8920145Z 2022-12-01T10:40:42.8920206Z OK 2022-12-01T10:40:42.8920211Z 2022-12-01T10:40:42.8920298Z Generating XML reports... 2022-12-01T10:40:42.8920640Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102433.xml 2022-12-01T10:40:42.8920917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8921107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8921389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8921511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8921695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_nf7gu2 2022-12-01T10:40:42.8921889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_nf7gu2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8922187Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8922192Z 2022-12-01T10:40:42.8922271Z Running tests... 2022-12-01T10:40:42.8922469Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8922736Z test_async_record_function_double_end_callbacks_new_signatures (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16882 2022-12-01T10:40:42.8922896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16883 2022-12-01T10:40:42.8923039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16884 2022-12-01T10:40:42.8923197Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16885 2022-12-01T10:40:42.8923469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8923596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8923879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8924019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8924201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ar1yvm1 2022-12-01T10:40:42.8924399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ar1yvm1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8924672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8924783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8925062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8925199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8925382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ibpqio8 2022-12-01T10:40:42.8925576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ibpqio8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8925846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8925972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8926252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8926387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8926558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2rvzsqd 2022-12-01T10:40:42.8926752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2rvzsqd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8927024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8927146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8927425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8927559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8927851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_350jxo 2022-12-01T10:40:42.8928049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_350jxo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8928201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8928368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8928591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8928755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8928825Z ok (3.142s) 2022-12-01T10:40:42.8928829Z 2022-12-01T10:40:42.8929030Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8929108Z Ran 1 test in 3.142s 2022-12-01T10:40:42.8929113Z 2022-12-01T10:40:42.8929174Z OK 2022-12-01T10:40:42.8929179Z 2022-12-01T10:40:42.8929253Z Generating XML reports... 2022-12-01T10:40:42.8929598Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102438.xml 2022-12-01T10:40:42.8929871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8929995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8930277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8930412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8930596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_oxd_o5 2022-12-01T10:40:42.8930791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_oxd_o5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8931051Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8931056Z 2022-12-01T10:40:42.8931116Z Running tests... 2022-12-01T10:40:42.8931317Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8931542Z test_async_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17105 2022-12-01T10:40:42.8931700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17106 2022-12-01T10:40:42.8931858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17107 2022-12-01T10:40:42.8932015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17108 2022-12-01T10:40:42.8932287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8932410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8932693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8932815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8933001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb46a0lw 2022-12-01T10:40:42.8933194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb46a0lw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8933467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8933591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8933871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8934007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8934225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk02a9bao 2022-12-01T10:40:42.8934407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk02a9bao/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8934682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8934805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8935081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8935245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8935428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ppqkr5v 2022-12-01T10:40:42.8935623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ppqkr5v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8935895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8936018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8936282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8936416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8936599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7s5s4lt 2022-12-01T10:40:42.8936796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7s5s4lt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8936960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8937125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8937289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8937454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8937509Z ok (2.142s) 2022-12-01T10:40:42.8937528Z 2022-12-01T10:40:42.8937711Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8937786Z Ran 1 test in 2.142s 2022-12-01T10:40:42.8937791Z 2022-12-01T10:40:42.8937853Z OK 2022-12-01T10:40:42.8937858Z 2022-12-01T10:40:42.8937942Z Generating XML reports... 2022-12-01T10:40:42.8938288Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102442.xml 2022-12-01T10:40:42.8938563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8938688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8938970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8939094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8939277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe_wnj3w 2022-12-01T10:40:42.8939470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe_wnj3w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8939724Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8939732Z 2022-12-01T10:40:42.8939805Z Running tests... 2022-12-01T10:40:42.8940004Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8940238Z test_async_static_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17328 2022-12-01T10:40:42.8940395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17329 2022-12-01T10:40:42.8940537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17330 2022-12-01T10:40:42.8940728Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17331 2022-12-01T10:40:42.8941004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8941127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8941409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8941593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8941780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3csr88q 2022-12-01T10:40:42.8942052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8942177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8942446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8942581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8942766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx0krfwo 2022-12-01T10:40:42.8942960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3csr88q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8943150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx0krfwo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8943424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8943548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8943826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8943963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8944133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_y3ijlz 2022-12-01T10:40:42.8944325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_y3ijlz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8944593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8944715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8944996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8945129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8945311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hvf0brr 2022-12-01T10:40:42.8945504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hvf0brr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8945657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8945819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8945980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8946142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8946215Z ok (2.239s) 2022-12-01T10:40:42.8946220Z 2022-12-01T10:40:42.8946420Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8946497Z Ran 1 test in 2.240s 2022-12-01T10:40:42.8946501Z 2022-12-01T10:40:42.8946563Z OK 2022-12-01T10:40:42.8946567Z 2022-12-01T10:40:42.8946639Z Generating XML reports... 2022-12-01T10:40:42.8946979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102446.xml 2022-12-01T10:40:42.8947290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8947417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8947696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8947832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8948047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xg_cbpo 2022-12-01T10:40:42.8948244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xg_cbpo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8948500Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8948505Z 2022-12-01T10:40:42.8948565Z Running tests... 2022-12-01T10:40:42.8948761Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8948992Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17551 2022-12-01T10:40:42.8949150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17552 2022-12-01T10:40:42.8949307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17553 2022-12-01T10:40:42.8949460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17554 2022-12-01T10:40:42.8949742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8949867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8950149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8950271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8950460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszqn15ve 2022-12-01T10:40:42.8950657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszqn15ve/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8950930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8951054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8951338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8951474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8951658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jqh8730 2022-12-01T10:40:42.8951838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jqh8730/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8952114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8952238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8952520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8952655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8952838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpew8o_3eg 2022-12-01T10:40:42.8953034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpew8o_3eg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8953307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8953431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8953725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8953864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8954047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5at86mgk 2022-12-01T10:40:42.8954241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5at86mgk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8954407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8954601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8954765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8954928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8954984Z ok (1.741s) 2022-12-01T10:40:42.8955002Z 2022-12-01T10:40:42.8955188Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8955267Z Ran 1 test in 1.742s 2022-12-01T10:40:42.8955273Z 2022-12-01T10:40:42.8955334Z OK 2022-12-01T10:40:42.8955339Z 2022-12-01T10:40:42.8955424Z Generating XML reports... 2022-12-01T10:40:42.8955766Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102450.xml 2022-12-01T10:40:42.8956038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8956167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8956447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8956570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8956757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_b62qgw 2022-12-01T10:40:42.8956955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_b62qgw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8957211Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8957216Z 2022-12-01T10:40:42.8957291Z Running tests... 2022-12-01T10:40:42.8957488Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8957711Z test_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17606 2022-12-01T10:40:42.8957872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17607 2022-12-01T10:40:42.8958013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17608 2022-12-01T10:40:42.8958169Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17609 2022-12-01T10:40:42.8958615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8958744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8959026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8959163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8959349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82ckycmi 2022-12-01T10:40:42.8959551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82ckycmi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8959823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8959933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8960214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8960409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8960596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmn1_6ktz 2022-12-01T10:40:42.8960790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmn1_6ktz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8961063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8961243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8961524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8961659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8961828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_v_igqc 2022-12-01T10:40:42.8962104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8962227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8962507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8962642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8962828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm355luyo 2022-12-01T10:40:42.8963024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_v_igqc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8963213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm355luyo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8963365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8963531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8963699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8963861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8963930Z ok (2.141s) 2022-12-01T10:40:42.8963935Z 2022-12-01T10:40:42.8964134Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8964212Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8964217Z 2022-12-01T10:40:42.8964281Z OK 2022-12-01T10:40:42.8964286Z 2022-12-01T10:40:42.8964357Z Generating XML reports... 2022-12-01T10:40:42.8964698Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102453.xml 2022-12-01T10:40:42.8964972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8965096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8965377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8965515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8965699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jv9fxsy 2022-12-01T10:40:42.8965895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jv9fxsy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8966153Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8966158Z 2022-12-01T10:40:42.8966218Z Running tests... 2022-12-01T10:40:42.8966418Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8966640Z test_builtin_remote_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17829 2022-12-01T10:40:42.8966830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17830 2022-12-01T10:40:42.8966992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17831 2022-12-01T10:40:42.8967144Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17832 2022-12-01T10:40:42.8967418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8967541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8967919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8968043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8968229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx75411hc 2022-12-01T10:40:42.8968424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx75411hc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8968706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8968829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8969111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8969247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8969431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y8kyv66 2022-12-01T10:40:42.8969614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y8kyv66/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8969886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8970010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8970292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8970432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8970617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2170xx0v 2022-12-01T10:40:42.8970812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2170xx0v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8971084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8971208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8971473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8971608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8971793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_stx0llq 2022-12-01T10:40:42.8971989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_stx0llq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8972155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8972320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8972483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8972647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8972703Z ok (2.038s) 2022-12-01T10:40:42.8972721Z 2022-12-01T10:40:42.8972910Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8972985Z Ran 1 test in 2.039s 2022-12-01T10:40:42.8972989Z 2022-12-01T10:40:42.8973051Z OK 2022-12-01T10:40:42.8973056Z 2022-12-01T10:40:42.8973142Z Generating XML reports... 2022-12-01T10:40:42.8973542Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102456.xml 2022-12-01T10:40:42.8973817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8973942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8974224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8974376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8974561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76k_n426 2022-12-01T10:40:42.8974759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76k_n426/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8975016Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8975021Z 2022-12-01T10:40:42.8975097Z Running tests... 2022-12-01T10:40:42.8975297Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8975425Z test_call_method_on_rref (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.8975663Z Tests that it is possible to call an instance method on a remote objet ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18052 2022-12-01T10:40:42.8975809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18053 2022-12-01T10:40:42.8975970Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18054 2022-12-01T10:40:42.8976124Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18055 2022-12-01T10:40:42.8976402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8976528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8976814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8976952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8977142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0dvuf1z 2022-12-01T10:40:42.8977341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0dvuf1z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8977601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8977728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8978007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8978142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8978329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0y9wycx0 2022-12-01T10:40:42.8978524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0y9wycx0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8978793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8978916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8979183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8979324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8979508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc6r91_v 2022-12-01T10:40:42.8979702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc6r91_v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8979972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8980131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8980414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8980548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8980733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlnyzx4l 2022-12-01T10:40:42.8980944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlnyzx4l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8981110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8981275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8981436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8981601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8981669Z ok (2.140s) 2022-12-01T10:40:42.8981674Z 2022-12-01T10:40:42.8981875Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8981951Z Ran 1 test in 2.141s 2022-12-01T10:40:42.8981956Z 2022-12-01T10:40:42.8982004Z OK 2022-12-01T10:40:42.8982022Z 2022-12-01T10:40:42.8982093Z Generating XML reports... 2022-12-01T10:40:42.8982433Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102500.xml 2022-12-01T10:40:42.8982709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8982833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8983113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8983250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8983435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gng26ya 2022-12-01T10:40:42.8983631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gng26ya/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8983876Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8983895Z 2022-12-01T10:40:42.8983957Z Running tests... 2022-12-01T10:40:42.8984155Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8984378Z test_callback_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18275 2022-12-01T10:40:42.8984534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18276 2022-12-01T10:40:42.8984690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18277 2022-12-01T10:40:42.8984845Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18278 2022-12-01T10:40:42.8985122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8985246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8985518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8985656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8985842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj3jr6uv 2022-12-01T10:40:42.8986040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj3jr6uv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8986313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8986480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8986764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8986901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8987073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5srl9cvn 2022-12-01T10:40:42.8987269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5srl9cvn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8987568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8987691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8987970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8988105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8988293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0wqzv1p 2022-12-01T10:40:42.8988489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0wqzv1p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8988760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8988870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8989153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8989288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8989471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f48jy6y 2022-12-01T10:40:42.8989667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f48jy6y/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8989834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8990000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8990162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8990310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8990380Z ok (2.239s) 2022-12-01T10:40:42.8990387Z 2022-12-01T10:40:42.8990584Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8990661Z Ran 1 test in 2.240s 2022-12-01T10:40:42.8990666Z 2022-12-01T10:40:42.8990727Z OK 2022-12-01T10:40:42.8990731Z 2022-12-01T10:40:42.8990817Z Generating XML reports... 2022-12-01T10:40:42.8991156Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102503.xml 2022-12-01T10:40:42.8991432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8991558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8991823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8991958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8992143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7vii376 2022-12-01T10:40:42.8992341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7vii376/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8992593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.8992598Z 2022-12-01T10:40:42.8992672Z Running tests... 2022-12-01T10:40:42.8992868Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8993117Z test_callback_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18498 2022-12-01T10:40:42.8993263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18499 2022-12-01T10:40:42.8993419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18500 2022-12-01T10:40:42.8993573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18501 2022-12-01T10:40:42.8993878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8994001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8994284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8994420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8994661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw631p1hf 2022-12-01T10:40:42.8994856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw631p1hf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8995117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8995241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8995520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8995656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8995840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0r6zgr3n 2022-12-01T10:40:42.8996034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0r6zgr3n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8996307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8996429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8996693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8996828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8997010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbc_upmf 2022-12-01T10:40:42.8997209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbc_upmf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8997481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.8997603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.8997882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.8998019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.8998378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypwy24or 2022-12-01T10:40:42.8998562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypwy24or/_remote_module_non_scriptable.py 2022-12-01T10:40:42.8998726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.8998894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.8999058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.8999220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.8999289Z ok (2.143s) 2022-12-01T10:40:42.8999294Z 2022-12-01T10:40:42.8999499Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.8999574Z Ran 1 test in 2.143s 2022-12-01T10:40:42.8999636Z 2022-12-01T10:40:42.8999686Z OK 2022-12-01T10:40:42.8999703Z 2022-12-01T10:40:42.8999775Z Generating XML reports... 2022-12-01T10:40:42.9000114Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102507.xml 2022-12-01T10:40:42.9000388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9000552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9000832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9000968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9001153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqyq58lv 2022-12-01T10:40:42.9001352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqyq58lv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9001597Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9001602Z 2022-12-01T10:40:42.9001677Z Running tests... 2022-12-01T10:40:42.9001874Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9002095Z test_callback_multi (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18721 2022-12-01T10:40:42.9002256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18722 2022-12-01T10:40:42.9002414Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18723 2022-12-01T10:40:42.9002568Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18724 2022-12-01T10:40:42.9002846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9002959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9003241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9003378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9003565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc7wmcw1 2022-12-01T10:40:42.9003764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc7wmcw1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9004041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9004165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9004445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9004580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9004756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszml0zo2 2022-12-01T10:40:42.9004954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszml0zo2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9005227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9005351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9005633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9005770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9005953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ju8zfaa 2022-12-01T10:40:42.9006148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ju8zfaa/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9006454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9006566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9006848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9006981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9007193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kgq3bct 2022-12-01T10:40:42.9007390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kgq3bct/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9007555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9007785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9007954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9008101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9008175Z ok (2.039s) 2022-12-01T10:40:42.9008180Z 2022-12-01T10:40:42.9008387Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9008464Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9008469Z 2022-12-01T10:40:42.9008531Z OK 2022-12-01T10:40:42.9008536Z 2022-12-01T10:40:42.9008624Z Generating XML reports... 2022-12-01T10:40:42.9008969Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102511.xml 2022-12-01T10:40:42.9009245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9009372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9009640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9009777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9009963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1j9tzdvj 2022-12-01T10:40:42.9010159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1j9tzdvj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9010416Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9010424Z 2022-12-01T10:40:42.9010498Z Running tests... 2022-12-01T10:40:42.9010695Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9010913Z test_callback_none (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18944 2022-12-01T10:40:42.9011060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18945 2022-12-01T10:40:42.9011218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18946 2022-12-01T10:40:42.9011371Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18947 2022-12-01T10:40:42.9011646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9011771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9012053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9012192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9012377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ldrqu27 2022-12-01T10:40:42.9012571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ldrqu27/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9012879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9013006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9013287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9013420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9013605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi5s1evi 2022-12-01T10:40:42.9013833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi5s1evi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9014107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9014231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9014495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9014633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9014816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0k74cep 2022-12-01T10:40:42.9015009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0k74cep/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9015281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9015407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9015689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9015823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9016009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz005qzh5 2022-12-01T10:40:42.9016191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz005qzh5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9016355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9016518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9016714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9016926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9017023Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.9017148Z ValueError('Expected error') 2022-12-01T10:40:42.9017240Z Traceback (most recent call last): 2022-12-01T10:40:42.9017491Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9017627Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9017914Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9018011Z raise ValueError(expected_err) 2022-12-01T10:40:42.9018102Z ValueError: Expected error 2022-12-01T10:40:42.9018109Z 2022-12-01T10:40:42.9018199Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.9018319Z ValueError('Expected error') 2022-12-01T10:40:42.9018398Z Traceback (most recent call last): 2022-12-01T10:40:42.9018657Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9018792Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9019098Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9019191Z raise ValueError(expected_err) 2022-12-01T10:40:42.9019276Z ValueError: Expected error 2022-12-01T10:40:42.9019281Z 2022-12-01T10:40:42.9019370Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.9019542Z ValueError('Expected error') 2022-12-01T10:40:42.9019643Z Traceback (most recent call last): 2022-12-01T10:40:42.9019899Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9020030Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9020307Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9020428Z raise ValueError(expected_err) 2022-12-01T10:40:42.9020513Z ValueError: Expected error 2022-12-01T10:40:42.9020518Z 2022-12-01T10:40:42.9020607Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.9020726Z ValueError('Expected error') 2022-12-01T10:40:42.9020802Z Traceback (most recent call last): 2022-12-01T10:40:42.9021054Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9021186Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9021463Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9021556Z raise ValueError(expected_err) 2022-12-01T10:40:42.9021641Z ValueError: Expected error 2022-12-01T10:40:42.9021646Z 2022-12-01T10:40:42.9021715Z ok (2.144s) 2022-12-01T10:40:42.9021720Z 2022-12-01T10:40:42.9021921Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9021986Z Ran 1 test in 2.144s 2022-12-01T10:40:42.9022005Z 2022-12-01T10:40:42.9022052Z OK 2022-12-01T10:40:42.9022057Z 2022-12-01T10:40:42.9022143Z Generating XML reports... 2022-12-01T10:40:42.9022484Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102514.xml 2022-12-01T10:40:42.9022761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9022884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9023166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9023305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9023492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoff1tbnd 2022-12-01T10:40:42.9023679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoff1tbnd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9023937Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9023943Z 2022-12-01T10:40:42.9024015Z Running tests... 2022-12-01T10:40:42.9024214Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9024437Z test_callback_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19167 2022-12-01T10:40:42.9024597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19168 2022-12-01T10:40:42.9024753Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19169 2022-12-01T10:40:42.9024906Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19170 2022-12-01T10:40:42.9025168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9025295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9025576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9025712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9025899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodp309hv 2022-12-01T10:40:42.9026125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodp309hv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9026402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9026528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9026810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9026961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9027145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n67wm7t 2022-12-01T10:40:42.9027340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n67wm7t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9027613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9027741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9028021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9028156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9028343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugeh6th6 2022-12-01T10:40:42.9028526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugeh6th6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9028802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9028925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9029205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9029344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9029528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7i4xzsuy 2022-12-01T10:40:42.9029721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7i4xzsuy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9029886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9030050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9030202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9030368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9030436Z ok (2.040s) 2022-12-01T10:40:42.9030442Z 2022-12-01T10:40:42.9030643Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9030719Z Ran 1 test in 2.040s 2022-12-01T10:40:42.9030724Z 2022-12-01T10:40:42.9030787Z OK 2022-12-01T10:40:42.9030794Z 2022-12-01T10:40:42.9030878Z Generating XML reports... 2022-12-01T10:40:42.9031223Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102518.xml 2022-12-01T10:40:42.9031481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9031606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9031889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9032025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9032213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zpbpmpf 2022-12-01T10:40:42.9032411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zpbpmpf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9032722Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9032728Z 2022-12-01T10:40:42.9032801Z Running tests... 2022-12-01T10:40:42.9032999Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9033211Z test_callback_with_error (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19390 2022-12-01T10:40:42.9033370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19391 2022-12-01T10:40:42.9033553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19392 2022-12-01T10:40:42.9033706Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19393 2022-12-01T10:40:42.9033979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9034103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9034388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9034528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9034717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_8oi3jp 2022-12-01T10:40:42.9034898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_8oi3jp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9035172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9035296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9035576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9035711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9035900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxpprb2s 2022-12-01T10:40:42.9036096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxpprb2s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9036366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9036475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9036755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9036894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9037079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyuosxkll 2022-12-01T10:40:42.9037274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyuosxkll/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9037549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9037674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9037952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9038089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9038412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1v_flhfr 2022-12-01T10:40:42.9038611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1v_flhfr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9038777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9038943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9039108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9039327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9039423Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.9039547Z ValueError('Expected error') 2022-12-01T10:40:42.9039629Z Traceback (most recent call last): 2022-12-01T10:40:42.9039894Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9040032Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9040357Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9040452Z raise ValueError(expected_err) 2022-12-01T10:40:42.9040541Z ValueError: Expected error 2022-12-01T10:40:42.9040546Z 2022-12-01T10:40:42.9040637Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.9040757Z ValueError('Expected error') 2022-12-01T10:40:42.9040835Z Traceback (most recent call last): 2022-12-01T10:40:42.9041095Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9041235Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9041516Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9041610Z raise ValueError(expected_err) 2022-12-01T10:40:42.9041698Z ValueError: Expected error 2022-12-01T10:40:42.9041703Z 2022-12-01T10:40:42.9041794Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.9041899Z ValueError('Expected error') 2022-12-01T10:40:42.9041992Z Traceback (most recent call last): 2022-12-01T10:40:42.9042246Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9042379Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9042660Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9042753Z raise ValueError(expected_err) 2022-12-01T10:40:42.9042838Z ValueError: Expected error 2022-12-01T10:40:42.9042842Z 2022-12-01T10:40:42.9042931Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.9043036Z ValueError('Expected error') 2022-12-01T10:40:42.9043129Z Traceback (most recent call last): 2022-12-01T10:40:42.9043381Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9043515Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9043792Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9043884Z raise ValueError(expected_err) 2022-12-01T10:40:42.9043969Z ValueError: Expected error 2022-12-01T10:40:42.9043974Z 2022-12-01T10:40:42.9044041Z ok (2.040s) 2022-12-01T10:40:42.9044046Z 2022-12-01T10:40:42.9044232Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9044310Z Ran 1 test in 2.040s 2022-12-01T10:40:42.9044315Z 2022-12-01T10:40:42.9044376Z OK 2022-12-01T10:40:42.9044381Z 2022-12-01T10:40:42.9044466Z Generating XML reports... 2022-12-01T10:40:42.9044807Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102521.xml 2022-12-01T10:40:42.9045085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9045212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9045492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9045628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9045833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25wzeqh6 2022-12-01T10:40:42.9046032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25wzeqh6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9046290Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9046295Z 2022-12-01T10:40:42.9046369Z Running tests... 2022-12-01T10:40:42.9046567Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9046815Z test_callback_with_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19613 2022-12-01T10:40:42.9046973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19614 2022-12-01T10:40:42.9047132Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19615 2022-12-01T10:40:42.9047271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19616 2022-12-01T10:40:42.9047552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9047739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9048033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9048172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9048359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs1irbno 2022-12-01T10:40:42.9048561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs1irbno/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9048834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9048959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9049231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9049369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9049552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntsl_ibi 2022-12-01T10:40:42.9049749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntsl_ibi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9050022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9050147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9050428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9050563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9050731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e_x5p9g 2022-12-01T10:40:42.9050927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e_x5p9g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9051199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9051325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9051604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9051742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9051928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgh5u3685 2022-12-01T10:40:42.9052122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgh5u3685/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9052287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9052475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9052643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9052803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9052873Z ok (2.041s) 2022-12-01T10:40:42.9052878Z 2022-12-01T10:40:42.9053076Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9053180Z Ran 1 test in 2.041s 2022-12-01T10:40:42.9053185Z 2022-12-01T10:40:42.9053248Z OK 2022-12-01T10:40:42.9053253Z 2022-12-01T10:40:42.9053337Z Generating XML reports... 2022-12-01T10:40:42.9053666Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102525.xml 2022-12-01T10:40:42.9053939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9054066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9054345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9054480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9054665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4naua3bs 2022-12-01T10:40:42.9054863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4naua3bs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9055121Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9055126Z 2022-12-01T10:40:42.9055199Z Running tests... 2022-12-01T10:40:42.9055382Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9055610Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19836 2022-12-01T10:40:42.9055769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19837 2022-12-01T10:40:42.9055926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19838 2022-12-01T10:40:42.9056081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19839 2022-12-01T10:40:42.9056357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9056484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9056767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9056903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9057076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqxykqys 2022-12-01T10:40:42.9057275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqxykqys/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9057550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9057673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9057954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9058093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9058279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv3gr6vu 2022-12-01T10:40:42.9058476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv3gr6vu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9058734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9058858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9059169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9059305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9059487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97tttsjw 2022-12-01T10:40:42.9059680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97tttsjw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9059976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9060100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9060378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9060498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9060686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpjsx262 2022-12-01T10:40:42.9060880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpjsx262/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9061046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9061212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9061377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9061541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9061611Z ok (2.142s) 2022-12-01T10:40:42.9061616Z 2022-12-01T10:40:42.9061800Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9061876Z Ran 1 test in 2.143s 2022-12-01T10:40:42.9061881Z 2022-12-01T10:40:42.9061942Z OK 2022-12-01T10:40:42.9061947Z 2022-12-01T10:40:42.9062034Z Generating XML reports... 2022-12-01T10:40:42.9062377Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102528.xml 2022-12-01T10:40:42.9062650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9062773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9063052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9063210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9063381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tld_yn0 2022-12-01T10:40:42.9063575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tld_yn0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9063831Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9063836Z 2022-12-01T10:40:42.9063910Z Running tests... 2022-12-01T10:40:42.9064108Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9064338Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20059 2022-12-01T10:40:42.9064496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20060 2022-12-01T10:40:42.9064656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20061 2022-12-01T10:40:42.9064797Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20062 2022-12-01T10:40:42.9065073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9065197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9065523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9065662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9065849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdk5fw9s 2022-12-01T10:40:42.9066048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdk5fw9s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9066320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9066470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9066738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9066875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9067062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyqaeuov 2022-12-01T10:40:42.9067259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyqaeuov/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9067532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9067655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9067937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9068076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9068245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yk3_nrj 2022-12-01T10:40:42.9068439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yk3_nrj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9068712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9068836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9069117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9069253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9069438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapogavof 2022-12-01T10:40:42.9069637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapogavof/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9069802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9069956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9070120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9070282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9070352Z ok (2.038s) 2022-12-01T10:40:42.9070357Z 2022-12-01T10:40:42.9070554Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9070630Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9070635Z 2022-12-01T10:40:42.9070697Z OK 2022-12-01T10:40:42.9070702Z 2022-12-01T10:40:42.9070789Z Generating XML reports... 2022-12-01T10:40:42.9071116Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102532.xml 2022-12-01T10:40:42.9071392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9071518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9071797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9071961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9072149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu422r5d_ 2022-12-01T10:40:42.9072344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu422r5d_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9072599Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9072605Z 2022-12-01T10:40:42.9072708Z Running tests... 2022-12-01T10:40:42.9072892Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9073130Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20282 2022-12-01T10:40:42.9073287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20283 2022-12-01T10:40:42.9073443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20284 2022-12-01T10:40:42.9073599Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20285 2022-12-01T10:40:42.9073873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9073997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9074277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9074416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9074587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpez4y_6u7 2022-12-01T10:40:42.9074781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpez4y_6u7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9075052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9075179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9075461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9075596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9075780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp651k101b 2022-12-01T10:40:42.9075977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp651k101b/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9076237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9076362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9076642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9076780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9076965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9eyvbe22 2022-12-01T10:40:42.9077158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9eyvbe22/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9077431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9077555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9077836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9077958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9078142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfilsw_9h 2022-12-01T10:40:42.9078469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfilsw_9h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9078694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9078861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9079025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9079188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9079293Z ok (1.736s) 2022-12-01T10:40:42.9079298Z 2022-12-01T10:40:42.9079486Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9079564Z Ran 1 test in 1.736s 2022-12-01T10:40:42.9079569Z 2022-12-01T10:40:42.9079632Z OK 2022-12-01T10:40:42.9079637Z 2022-12-01T10:40:42.9079723Z Generating XML reports... 2022-12-01T10:40:42.9080064Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102535.xml 2022-12-01T10:40:42.9080343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9080468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9080750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9080886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9081062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzo5nid4r 2022-12-01T10:40:42.9081259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzo5nid4r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9081514Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9081519Z 2022-12-01T10:40:42.9081594Z Running tests... 2022-12-01T10:40:42.9081792Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9082005Z test_deadlock (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20337 2022-12-01T10:40:42.9082164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20338 2022-12-01T10:40:42.9082321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20339 2022-12-01T10:40:42.9082463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20340 2022-12-01T10:40:42.9082741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9082867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9083150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9083286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9083474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14i8t51l 2022-12-01T10:40:42.9083669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14i8t51l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9083942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9084067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9084332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9084471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9084656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm3qawll 2022-12-01T10:40:42.9084850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm3qawll/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9085152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9085277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9085559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9085694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9085864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbluqf2v 2022-12-01T10:40:42.9086086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbluqf2v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9086357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9086479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9086757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9086892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9087076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdewb5ji 2022-12-01T10:40:42.9087270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdewb5ji/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9087437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9087589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9087811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9087975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9088158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9088340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9088516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9088691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9088998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9089301Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9089581Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9089873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9089944Z ok (3.140s) 2022-12-01T10:40:42.9089949Z 2022-12-01T10:40:42.9090151Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9090229Z Ran 1 test in 3.140s 2022-12-01T10:40:42.9090234Z 2022-12-01T10:40:42.9090296Z OK 2022-12-01T10:40:42.9090300Z 2022-12-01T10:40:42.9090385Z Generating XML reports... 2022-12-01T10:40:42.9090730Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102538.xml 2022-12-01T10:40:42.9091004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9091118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9091401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9091539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9091788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71bam8y6 2022-12-01T10:40:42.9091985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71bam8y6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9092241Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9092247Z 2022-12-01T10:40:42.9092321Z Running tests... 2022-12-01T10:40:42.9092523Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9092751Z test_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20572 2022-12-01T10:40:42.9092910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20573 2022-12-01T10:40:42.9093065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20574 2022-12-01T10:40:42.9093218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20575 2022-12-01T10:40:42.9093496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9093620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9093903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9094039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9094225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmt4f0t7l 2022-12-01T10:40:42.9094411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmt4f0t7l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9094684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9094810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9095093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9095230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9095414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1g4v8upy 2022-12-01T10:40:42.9095611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1g4v8upy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9095885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9095998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9096279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9096413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9096599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwj2z8pr 2022-12-01T10:40:42.9096797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwj2z8pr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9097068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9097192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9097472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9097612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9097781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hawryf1 2022-12-01T10:40:42.9097975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hawryf1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9098141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9098338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9098501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9098664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9098732Z ok (2.139s) 2022-12-01T10:40:42.9098738Z 2022-12-01T10:40:42.9098940Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9099029Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9099047Z 2022-12-01T10:40:42.9099095Z OK 2022-12-01T10:40:42.9099099Z 2022-12-01T10:40:42.9099187Z Generating XML reports... 2022-12-01T10:40:42.9099528Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102543.xml 2022-12-01T10:40:42.9099801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9099928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9100206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9100342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9100527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47mmvjrb 2022-12-01T10:40:42.9100710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47mmvjrb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9100969Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9100975Z 2022-12-01T10:40:42.9101048Z Running tests... 2022-12-01T10:40:42.9101247Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9101374Z test_default_timeout_used (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.9101614Z Tests that if no timeout is passed into rpc_async and rpc_sync, then the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20799 2022-12-01T10:40:42.9101772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20800 2022-12-01T10:40:42.9101928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20801 2022-12-01T10:40:42.9102068Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20802 2022-12-01T10:40:42.9102343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9102470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9102753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9102887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9103074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6f14q6li 2022-12-01T10:40:42.9103270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6f14q6li/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9103543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9103667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9103931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9104070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9104256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnk9k0p1 2022-12-01T10:40:42.9104451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnk9k0p1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9104720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9104877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9105156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9105291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9105461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq53k_yv0 2022-12-01T10:40:42.9105685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq53k_yv0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9105957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9106081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9106360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9106496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9106677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qa9izx8 2022-12-01T10:40:42.9106871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qa9izx8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9107035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9107187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9107352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9107514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9107583Z ok (4.142s) 2022-12-01T10:40:42.9107588Z 2022-12-01T10:40:42.9107786Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9107862Z Ran 1 test in 4.142s 2022-12-01T10:40:42.9107867Z 2022-12-01T10:40:42.9107931Z OK 2022-12-01T10:40:42.9107935Z 2022-12-01T10:40:42.9108021Z Generating XML reports... 2022-12-01T10:40:42.9108349Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102547.xml 2022-12-01T10:40:42.9108622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9108747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9109027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9109161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9109347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5l56jhw 2022-12-01T10:40:42.9109543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5l56jhw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9109803Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9109808Z 2022-12-01T10:40:42.9109883Z Running tests... 2022-12-01T10:40:42.9110068Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9110294Z test_disable_gil_profiling (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21022 2022-12-01T10:40:42.9110456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21023 2022-12-01T10:40:42.9110612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21024 2022-12-01T10:40:42.9110766Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21025 2022-12-01T10:40:42.9111041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9111165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9111490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9111628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9111802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixy6ifc4 2022-12-01T10:40:42.9111999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixy6ifc4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9112299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9112423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9112703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9112839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9113026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbvj6c_ju 2022-12-01T10:40:42.9113223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbvj6c_ju/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9113479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9113602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9113883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9114018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9114204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcuwy4t6f 2022-12-01T10:40:42.9114398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcuwy4t6f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9114672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9114796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9115077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9115198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9115383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ax3n67p 2022-12-01T10:40:42.9115580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ax3n67p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9115745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9115911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9116075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9116239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9116308Z ok (2.042s) 2022-12-01T10:40:42.9116312Z 2022-12-01T10:40:42.9116497Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9116572Z Ran 1 test in 2.042s 2022-12-01T10:40:42.9116577Z 2022-12-01T10:40:42.9116639Z OK 2022-12-01T10:40:42.9116644Z 2022-12-01T10:40:42.9116730Z Generating XML reports... 2022-12-01T10:40:42.9117074Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102552.xml 2022-12-01T10:40:42.9117349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9117474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9117785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9117924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9118100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9papalc 2022-12-01T10:40:42.9118446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9papalc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9118706Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9118768Z 2022-12-01T10:40:42.9118844Z Running tests... 2022-12-01T10:40:42.9119043Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9119267Z test_dist_init_decorator (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21245 2022-12-01T10:40:42.9119426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21246 2022-12-01T10:40:42.9119586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21247 2022-12-01T10:40:42.9119727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21248 2022-12-01T10:40:42.9120001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9120124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9120406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9120546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9120733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64_i90ee 2022-12-01T10:40:42.9120927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64_i90ee/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9121199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9121325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9121591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9121726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9121910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurug5gug 2022-12-01T10:40:42.9122110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurug5gug/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9122384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9122509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9122792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9122929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9123102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqupdychk 2022-12-01T10:40:42.9123302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqupdychk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9123573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9123700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9123980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9124114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9124300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpat8wv3 2022-12-01T10:40:42.9124538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpat8wv3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9124706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9124855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9125017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9125177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9125273Z ok (2.038s) 2022-12-01T10:40:42.9125278Z 2022-12-01T10:40:42.9125480Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9125556Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9125561Z 2022-12-01T10:40:42.9125622Z OK 2022-12-01T10:40:42.9125627Z 2022-12-01T10:40:42.9125712Z Generating XML reports... 2022-12-01T10:40:42.9126042Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102556.xml 2022-12-01T10:40:42.9126318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9126443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9126722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9126857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9127047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp508re6ud 2022-12-01T10:40:42.9127244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp508re6ud/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9127499Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9127504Z 2022-12-01T10:40:42.9127579Z Running tests... 2022-12-01T10:40:42.9127828Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9128054Z test_duplicate_name (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21468 2022-12-01T10:40:42.9128212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21469 2022-12-01T10:40:42.9128371Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21470 2022-12-01T10:40:42.9128525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21471 2022-12-01T10:40:42.9128809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9128933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9129215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9129355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9129529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2k148sq 2022-12-01T10:40:42.9129724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2k148sq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9129997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9130121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9130404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9130540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9130726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3odlr0h 2022-12-01T10:40:42.9130921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3odlr0h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9131216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9131344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9131627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9131761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9131947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsgms2v6 2022-12-01T10:40:42.9132202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsgms2v6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9132473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9132597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9132880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9133001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9133185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuc3iovt 2022-12-01T10:40:42.9133379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuc3iovt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9133544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9133711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9133874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9134036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9134105Z ok (1.839s) 2022-12-01T10:40:42.9134110Z 2022-12-01T10:40:42.9134300Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9134376Z Ran 1 test in 1.839s 2022-12-01T10:40:42.9134381Z 2022-12-01T10:40:42.9134445Z OK 2022-12-01T10:40:42.9134450Z 2022-12-01T10:40:42.9134535Z Generating XML reports... 2022-12-01T10:40:42.9134879Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102559.xml 2022-12-01T10:40:42.9135151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9135282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9135561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9135696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9135868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hrpxtvf 2022-12-01T10:40:42.9136068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hrpxtvf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9136322Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9136327Z 2022-12-01T10:40:42.9136402Z Running tests... 2022-12-01T10:40:42.9136599Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9136821Z test_duplicate_name_2 (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21603 2022-12-01T10:40:42.9136982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21604 2022-12-01T10:40:42.9137140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21605 2022-12-01T10:40:42.9137281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21606 2022-12-01T10:40:42.9137589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9137717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9138002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9138137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9138324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqfg3tp7 2022-12-01T10:40:42.9138551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqfg3tp7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9138823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9138949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9139215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9139353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9139539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodbz8c66 2022-12-01T10:40:42.9139809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9139932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9140213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9140351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9140537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdz9iwogf 2022-12-01T10:40:42.9140718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodbz8c66/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9140913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdz9iwogf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9141182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9141305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9141582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9141717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9141903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7j_xsd8 2022-12-01T10:40:42.9142098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7j_xsd8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9142262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9142411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9142578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9142739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9142808Z ok (1.839s) 2022-12-01T10:40:42.9142813Z 2022-12-01T10:40:42.9143013Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9143090Z Ran 1 test in 1.840s 2022-12-01T10:40:42.9143097Z 2022-12-01T10:40:42.9143160Z OK 2022-12-01T10:40:42.9143164Z 2022-12-01T10:40:42.9143250Z Generating XML reports... 2022-12-01T10:40:42.9143580Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102602.xml 2022-12-01T10:40:42.9143852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9143977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9144300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9144436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9144622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gtepqne 2022-12-01T10:40:42.9144820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gtepqne/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9145103Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9145109Z 2022-12-01T10:40:42.9145182Z Running tests... 2022-12-01T10:40:42.9145366Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9145585Z test_expected_src (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21738 2022-12-01T10:40:42.9145745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21739 2022-12-01T10:40:42.9145903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21740 2022-12-01T10:40:42.9146058Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21741 2022-12-01T10:40:42.9146332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9146456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9146740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9146876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9147050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptep75sg9 2022-12-01T10:40:42.9147248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptep75sg9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9147522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9147647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9147926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9148062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9148251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnww700f 2022-12-01T10:40:42.9148446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnww700f/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9148704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9148829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9149111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9149248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9149433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx2q__0l 2022-12-01T10:40:42.9149627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx2q__0l/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9149901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9150028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9150307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9150428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9150614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuy3sxo7 2022-12-01T10:40:42.9150841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuy3sxo7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9151006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9151170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9151333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9151525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9151594Z ok (2.038s) 2022-12-01T10:40:42.9151599Z 2022-12-01T10:40:42.9151787Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9151863Z Ran 1 test in 2.038s 2022-12-01T10:40:42.9151868Z 2022-12-01T10:40:42.9151931Z OK 2022-12-01T10:40:42.9151936Z 2022-12-01T10:40:42.9152021Z Generating XML reports... 2022-12-01T10:40:42.9152363Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102606.xml 2022-12-01T10:40:42.9152637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9152762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9153040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9153179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9153351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsyfiubj 2022-12-01T10:40:42.9153548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsyfiubj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9153803Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9153808Z 2022-12-01T10:40:42.9153885Z Running tests... 2022-12-01T10:40:42.9154082Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9154310Z test_function_not_on_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21961 2022-12-01T10:40:42.9154469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21962 2022-12-01T10:40:42.9154626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21963 2022-12-01T10:40:42.9154770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21964 2022-12-01T10:40:42.9155046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9155169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9155451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9155590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9155778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnep7_9hj 2022-12-01T10:40:42.9155973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnep7_9hj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9156244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9156372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9156637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9156770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9156953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnqc1e0j 2022-12-01T10:40:42.9157199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnqc1e0j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9157475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9157598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9157877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9158041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9158362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8rlpkwf 2022-12-01T10:40:42.9158637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9158762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9159044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9159178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9159361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56ag75yi 2022-12-01T10:40:42.9159556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8rlpkwf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9159744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56ag75yi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9159912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9160063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9160230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9160392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9160487Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.9161132Z AttributeError("Can't get attribute 'foo_add' on Default RPC pickler does not serialize\n function code. Ensure that UDFs are defined on both caller and\n callee modules.") 2022-12-01T10:40:42.9161226Z Traceback (most recent call last): 2022-12-01T10:40:42.9161492Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 159, in deserialize 2022-12-01T10:40:42.9161580Z ret = unpickler.load() 2022-12-01T10:40:42.9162014Z AttributeError: Can't get attribute 'foo_add' on 2022-12-01T10:40:42.9162020Z 2022-12-01T10:40:42.9162159Z The above exception was the direct cause of the following exception: 2022-12-01T10:40:42.9162165Z 2022-12-01T10:40:42.9162246Z Traceback (most recent call last): 2022-12-01T10:40:42.9162506Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 205, in _run_function 2022-12-01T10:40:42.9162583Z raise python_udf 2022-12-01T10:40:42.9163068Z AttributeError: Can't get attribute 'foo_add' on Default RPC pickler does not serialize 2022-12-01T10:40:42.9163206Z function code. Ensure that UDFs are defined on both caller and 2022-12-01T10:40:42.9163284Z callee modules. 2022-12-01T10:40:42.9163289Z 2022-12-01T10:40:42.9163358Z ok (2.239s) 2022-12-01T10:40:42.9163363Z 2022-12-01T10:40:42.9163563Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9163641Z Ran 1 test in 2.240s 2022-12-01T10:40:42.9163646Z 2022-12-01T10:40:42.9163746Z OK 2022-12-01T10:40:42.9163751Z 2022-12-01T10:40:42.9163837Z Generating XML reports... 2022-12-01T10:40:42.9164180Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102609.xml 2022-12-01T10:40:42.9164455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9164579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9164902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9165038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9165225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp628e6hra 2022-12-01T10:40:42.9165405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp628e6hra/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9165664Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9165670Z 2022-12-01T10:40:42.9165745Z Running tests... 2022-12-01T10:40:42.9165942Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9166160Z test_future_done (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22184 2022-12-01T10:40:42.9166323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22185 2022-12-01T10:40:42.9166482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22186 2022-12-01T10:40:42.9166636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22187 2022-12-01T10:40:42.9166912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9167024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9167310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9167446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9167631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2k3voddg 2022-12-01T10:40:42.9167921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2k3voddg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9168206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9168331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9168613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9168736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9168924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8ieqnuk 2022-12-01T10:40:42.9169123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8ieqnuk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9169400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9169525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9169807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9169942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9170126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_iy90ru 2022-12-01T10:40:42.9170319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_iy90ru/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9170619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9170746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9171026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9171160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9171344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5j49ibp 2022-12-01T10:40:42.9171565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5j49ibp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9171729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9171892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9172041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9172207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9172277Z ok (2.038s) 2022-12-01T10:40:42.9172282Z 2022-12-01T10:40:42.9172482Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9172558Z Ran 1 test in 2.038s 2022-12-01T10:40:42.9172563Z 2022-12-01T10:40:42.9172624Z OK 2022-12-01T10:40:42.9172629Z 2022-12-01T10:40:42.9172715Z Generating XML reports... 2022-12-01T10:40:42.9173057Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102613.xml 2022-12-01T10:40:42.9173330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9173441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9173724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9173858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9174044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ssej9ts 2022-12-01T10:40:42.9174240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ssej9ts/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9174495Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9174503Z 2022-12-01T10:40:42.9174579Z Running tests... 2022-12-01T10:40:42.9174779Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9174993Z test_future_done_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22407 2022-12-01T10:40:42.9175152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22408 2022-12-01T10:40:42.9175309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22409 2022-12-01T10:40:42.9175464Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22410 2022-12-01T10:40:42.9175739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9175863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9176144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9176283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9176470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwox0tv3o 2022-12-01T10:40:42.9176654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwox0tv3o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9176927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9177086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9177369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9177505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9177692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1azuwimr 2022-12-01T10:40:42.9177918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1azuwimr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9178190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9178315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9178581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9178719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9178903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4hvp175 2022-12-01T10:40:42.9179097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4hvp175/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9179368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9179494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9179774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9179912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9180082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpon3zu3vc 2022-12-01T10:40:42.9180280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpon3zu3vc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9180444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9180609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9180772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9180935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9181032Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.9181151Z ValueError('Expected error') 2022-12-01T10:40:42.9181230Z Traceback (most recent call last): 2022-12-01T10:40:42.9181491Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9181629Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9181916Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9182010Z raise ValueError(expected_err) 2022-12-01T10:40:42.9182098Z ValueError: Expected error 2022-12-01T10:40:42.9182105Z 2022-12-01T10:40:42.9182196Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.9182316Z ValueError('Expected error') 2022-12-01T10:40:42.9182393Z Traceback (most recent call last): 2022-12-01T10:40:42.9182649Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9182786Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9183067Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9183161Z raise ValueError(expected_err) 2022-12-01T10:40:42.9183249Z ValueError: Expected error 2022-12-01T10:40:42.9183254Z 2022-12-01T10:40:42.9183344Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.9183498Z ValueError('Expected error') 2022-12-01T10:40:42.9183578Z Traceback (most recent call last): 2022-12-01T10:40:42.9183833Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9183966Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9184244Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9184367Z raise ValueError(expected_err) 2022-12-01T10:40:42.9184452Z ValueError: Expected error 2022-12-01T10:40:42.9184456Z 2022-12-01T10:40:42.9184546Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.9184664Z ValueError('Expected error') 2022-12-01T10:40:42.9184741Z Traceback (most recent call last): 2022-12-01T10:40:42.9184995Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9185127Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9185411Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:42.9185502Z raise ValueError(expected_err) 2022-12-01T10:40:42.9185588Z ValueError: Expected error 2022-12-01T10:40:42.9185594Z 2022-12-01T10:40:42.9185662Z ok (2.240s) 2022-12-01T10:40:42.9185667Z 2022-12-01T10:40:42.9185867Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9185933Z Ran 1 test in 2.240s 2022-12-01T10:40:42.9185938Z 2022-12-01T10:40:42.9186000Z OK 2022-12-01T10:40:42.9186005Z 2022-12-01T10:40:42.9186089Z Generating XML reports... 2022-12-01T10:40:42.9186431Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102616.xml 2022-12-01T10:40:42.9186704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9186833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9187114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9187250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9187424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpid47r3ic 2022-12-01T10:40:42.9187626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpid47r3ic/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9187879Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9187884Z 2022-12-01T10:40:42.9187958Z Running tests... 2022-12-01T10:40:42.9188153Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9188371Z test_future_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22630 2022-12-01T10:40:42.9188531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22631 2022-12-01T10:40:42.9188687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22632 2022-12-01T10:40:42.9188842Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22633 2022-12-01T10:40:42.9189105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9189232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9189511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9189646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9189832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8ycmdv4 2022-12-01T10:40:42.9190064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8ycmdv4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9190340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9190464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9190730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9190897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9191083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4ja0h3a 2022-12-01T10:40:42.9191354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9191477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9191755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9191890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9192073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9aflx5hn 2022-12-01T10:40:42.9192267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4ja0h3a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9192445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9aflx5hn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9192719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9192842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9193119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9193253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9193438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaft6k2me 2022-12-01T10:40:42.9193633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaft6k2me/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9193798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9193950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9194116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9194277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9194345Z ok (2.139s) 2022-12-01T10:40:42.9194350Z 2022-12-01T10:40:42.9194549Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9194625Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9194630Z 2022-12-01T10:40:42.9194691Z OK 2022-12-01T10:40:42.9194695Z 2022-12-01T10:40:42.9194782Z Generating XML reports... 2022-12-01T10:40:42.9195118Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102620.xml 2022-12-01T10:40:42.9195379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9195502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9195783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9195919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9196104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphld1fgpo 2022-12-01T10:40:42.9196300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphld1fgpo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9196596Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9196602Z 2022-12-01T10:40:42.9196678Z Running tests... 2022-12-01T10:40:42.9196863Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9197095Z test_future_nested_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22853 2022-12-01T10:40:42.9197252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22854 2022-12-01T10:40:42.9197444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22855 2022-12-01T10:40:42.9197599Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22856 2022-12-01T10:40:42.9197872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9197998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9198438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9198577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9198753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lst82jp 2022-12-01T10:40:42.9198952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lst82jp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9199230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9199355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9199636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9199773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9199962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdfcbev49 2022-12-01T10:40:42.9200161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdfcbev49/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9200438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9200549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9200829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9200969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9201156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnb6epq8 2022-12-01T10:40:42.9201353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnb6epq8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9201627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9201756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9202036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9202209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9202394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyhjq338 2022-12-01T10:40:42.9202592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyhjq338/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9202756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9202922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9203087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9203329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9203403Z ok (2.039s) 2022-12-01T10:40:42.9203408Z 2022-12-01T10:40:42.9203608Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9203671Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9203676Z 2022-12-01T10:40:42.9203739Z OK 2022-12-01T10:40:42.9203743Z 2022-12-01T10:40:42.9203828Z Generating XML reports... 2022-12-01T10:40:42.9204168Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102624.xml 2022-12-01T10:40:42.9204489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9204616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9204896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9205037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9205210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_490g0n 2022-12-01T10:40:42.9205403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_490g0n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9205657Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9205662Z 2022-12-01T10:40:42.9205738Z Running tests... 2022-12-01T10:40:42.9205936Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9206616Z test_future_wait_twice (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69480 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.219s) 2022-12-01T10:40:42.9206625Z 2022-12-01T10:40:42.9206818Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9206894Z Ran 1 test in 0.219s 2022-12-01T10:40:42.9206899Z 2022-12-01T10:40:42.9206973Z OK (skipped=1) 2022-12-01T10:40:42.9206978Z 2022-12-01T10:40:42.9207063Z Generating XML reports... 2022-12-01T10:40:42.9207388Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102627.xml 2022-12-01T10:40:42.9207666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9207857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9208141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9208279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9208470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwi5c4rk 2022-12-01T10:40:42.9208669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwi5c4rk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9208925Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9208930Z 2022-12-01T10:40:42.9209005Z Running tests... 2022-12-01T10:40:42.9209187Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9209413Z test_get_worker_infos (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23086 2022-12-01T10:40:42.9209575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23087 2022-12-01T10:40:42.9209731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23088 2022-12-01T10:40:42.9209886Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23089 2022-12-01T10:40:42.9210206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9210335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9210616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9210738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9210955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjt_9968 2022-12-01T10:40:42.9211148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjt_9968/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9211418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9211541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9211823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9211963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9212147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsid_alk9 2022-12-01T10:40:42.9212342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsid_alk9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9212601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9212728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9213006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9213141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9213328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmn2m775 2022-12-01T10:40:42.9213525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmn2m775/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9213797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9213924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9214190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9214329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9214514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkg4uj9d2 2022-12-01T10:40:42.9214709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkg4uj9d2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9214877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9215045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9215208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9215373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9215441Z ok (2.139s) 2022-12-01T10:40:42.9215446Z 2022-12-01T10:40:42.9215632Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9215714Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9215719Z 2022-12-01T10:40:42.9215780Z OK 2022-12-01T10:40:42.9215784Z 2022-12-01T10:40:42.9215871Z Generating XML reports... 2022-12-01T10:40:42.9216210Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102629.xml 2022-12-01T10:40:42.9216484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9216641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9216922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9217045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9217230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_n_r4tl 2022-12-01T10:40:42.9217477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_n_r4tl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9217732Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9217738Z 2022-12-01T10:40:42.9217811Z Running tests... 2022-12-01T10:40:42.9218009Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9218160Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest) 2022-12-01T10:40:42.9218356Z Test graceful termination. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23309 2022-12-01T10:40:42.9218515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23310 2022-12-01T10:40:42.9218658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23311 2022-12-01T10:40:42.9218812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23312 2022-12-01T10:40:42.9219085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9219212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9219494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9219630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9219818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr6o27b4 2022-12-01T10:40:42.9220014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr6o27b4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9220275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9220399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9220683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9220822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9221008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6x1x77m4 2022-12-01T10:40:42.9221202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6x1x77m4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9221471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9221598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9221879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9222001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9222186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g3vpl83 2022-12-01T10:40:42.9222385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g3vpl83/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9222654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9222777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9223056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9223220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9223407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt72n3qck 2022-12-01T10:40:42.9223589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt72n3qck/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9223756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9223920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9224112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9224274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9224343Z ok (2.239s) 2022-12-01T10:40:42.9224348Z 2022-12-01T10:40:42.9224549Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9224625Z Ran 1 test in 2.240s 2022-12-01T10:40:42.9224632Z 2022-12-01T10:40:42.9224695Z OK 2022-12-01T10:40:42.9224699Z 2022-12-01T10:40:42.9224770Z Generating XML reports... 2022-12-01T10:40:42.9225113Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102632.xml 2022-12-01T10:40:42.9225390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9225517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9225796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9225932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9226120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyns9uc7i 2022-12-01T10:40:42.9226319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyns9uc7i/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9226562Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9226581Z 2022-12-01T10:40:42.9226642Z Running tests... 2022-12-01T10:40:42.9226839Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9227070Z test_handle_send_exceptions (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23532 2022-12-01T10:40:42.9227229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23533 2022-12-01T10:40:42.9227387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23534 2022-12-01T10:40:42.9227543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23535 2022-12-01T10:40:42.9227818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9227944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9228213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9228349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9228535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvef3ns3v 2022-12-01T10:40:42.9228731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvef3ns3v/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9229004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9229130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9229408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9229543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9229746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kpl0eb1 2022-12-01T10:40:42.9229945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kpl0eb1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9230213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9230337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9230647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9230781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9230965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprci10vdd 2022-12-01T10:40:42.9231162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprci10vdd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9231435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9231545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9231823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9231957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9232142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrl8_pf3 2022-12-01T10:40:42.9232336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrl8_pf3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9232501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9232667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9232833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9232997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9233162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9233339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9233514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9233817Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9233991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9234289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9234585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9234880Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9235147Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9235405Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.9235673Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.9235994Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9236289Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #1 to worker2: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:132) 2022-12-01T10:40:42.9236546Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9236829Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9237129Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #2 to worker2: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:132) 2022-12-01T10:40:42.9237198Z ok (2.139s) 2022-12-01T10:40:42.9237203Z 2022-12-01T10:40:42.9237399Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9237478Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9237483Z 2022-12-01T10:40:42.9237544Z OK 2022-12-01T10:40:42.9237549Z 2022-12-01T10:40:42.9237638Z Generating XML reports... 2022-12-01T10:40:42.9237978Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102636.xml 2022-12-01T10:40:42.9238407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9238535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9238818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9238956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9239132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81csqkxr 2022-12-01T10:40:42.9239330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81csqkxr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9239589Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9239594Z 2022-12-01T10:40:42.9239668Z Running tests... 2022-12-01T10:40:42.9239869Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9240093Z test_ignore_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23767 2022-12-01T10:40:42.9240254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23768 2022-12-01T10:40:42.9240411Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23769 2022-12-01T10:40:42.9240554Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23770 2022-12-01T10:40:42.9240831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9240955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9241238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9241377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9241567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptiayjtsg 2022-12-01T10:40:42.9241767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptiayjtsg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9242043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9242168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9242489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9242627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9242811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwo34pec 2022-12-01T10:40:42.9243009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwo34pec/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9243315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9243442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9243723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9243858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9244030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lc1m0av 2022-12-01T10:40:42.9244226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lc1m0av/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9244498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9244623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9244906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9245047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9245231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5fry4ho 2022-12-01T10:40:42.9245427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5fry4ho/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9245594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9245749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9245915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9246078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9246257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9246437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9246612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9246913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9247087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9247388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9247668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9248024Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9248581Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-12-01T10:40:42.9248823Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1) 2022-12-01T10:40:42.9248829Z 2022-12-01T10:40:42.9249366Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-12-01T10:40:42.9249580Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1) 2022-12-01T10:40:42.9249585Z 2022-12-01T10:40:42.9250117Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-12-01T10:40:42.9250299Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1) 2022-12-01T10:40:42.9250306Z 2022-12-01T10:40:42.9250833Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-12-01T10:40:42.9251016Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1) 2022-12-01T10:40:42.9251021Z 2022-12-01T10:40:42.9251091Z ok (2.136s) 2022-12-01T10:40:42.9251096Z 2022-12-01T10:40:42.9251300Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9251378Z Ran 1 test in 2.136s 2022-12-01T10:40:42.9251383Z 2022-12-01T10:40:42.9251446Z OK 2022-12-01T10:40:42.9251453Z 2022-12-01T10:40:42.9251539Z Generating XML reports... 2022-12-01T10:40:42.9251884Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102639.xml 2022-12-01T10:40:42.9252162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9252274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9252565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9252704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9252891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bd3ep4j 2022-12-01T10:40:42.9253091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bd3ep4j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9253349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9253356Z 2022-12-01T10:40:42.9253432Z Running tests... 2022-12-01T10:40:42.9253628Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9253847Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24002 2022-12-01T10:40:42.9253992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24003 2022-12-01T10:40:42.9254177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24004 2022-12-01T10:40:42.9254332Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24005 2022-12-01T10:40:42.9254608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9254731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9255014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9255179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9255366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnslb1li5 2022-12-01T10:40:42.9255549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnslb1li5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9255824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9255946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9256226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9256361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9256545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2gwvhkk 2022-12-01T10:40:42.9256741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2gwvhkk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9257010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9257132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9257401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9257535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9257718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskr7mc6o 2022-12-01T10:40:42.9257911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskr7mc6o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9258181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9258305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9258582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9258715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9258883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6u_mgir0 2022-12-01T10:40:42.9259075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6u_mgir0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9259239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9259403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9259562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9259723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9259902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9260079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9260251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9260569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9260741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9261038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9261333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9261649Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9261717Z ok (2.239s) 2022-12-01T10:40:42.9261722Z 2022-12-01T10:40:42.9261922Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9261996Z Ran 1 test in 2.240s 2022-12-01T10:40:42.9262001Z 2022-12-01T10:40:42.9262062Z OK 2022-12-01T10:40:42.9262066Z 2022-12-01T10:40:42.9262138Z Generating XML reports... 2022-12-01T10:40:42.9262480Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102643.xml 2022-12-01T10:40:42.9262753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9262878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9263158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9263297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9263483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzalc46mc 2022-12-01T10:40:42.9263678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzalc46mc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9263920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9263939Z 2022-12-01T10:40:42.9263999Z Running tests... 2022-12-01T10:40:42.9264195Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9264414Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24237 2022-12-01T10:40:42.9264569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24238 2022-12-01T10:40:42.9264729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24239 2022-12-01T10:40:42.9264881Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24240 2022-12-01T10:40:42.9265155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9265278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9265548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9265689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9265875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf6ym4n3 2022-12-01T10:40:42.9266073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf6ym4n3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9266345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9266471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9266752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9266888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9267071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpwy6fnh 2022-12-01T10:40:42.9267292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpwy6fnh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9267562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9267685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9267964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9268126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9268310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmth8k9r 2022-12-01T10:40:42.9268505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmth8k9r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9268774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9268886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9269163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9269298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9269480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8_c9e4n 2022-12-01T10:40:42.9269675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8_c9e4n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9269841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9270005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9270166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9270327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9270493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9270671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9270843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9271017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9271321Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9271616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9271912Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9272202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9272269Z ok (2.141s) 2022-12-01T10:40:42.9272274Z 2022-12-01T10:40:42.9272458Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9272533Z Ran 1 test in 2.142s 2022-12-01T10:40:42.9272538Z 2022-12-01T10:40:42.9272598Z OK 2022-12-01T10:40:42.9272603Z 2022-12-01T10:40:42.9272687Z Generating XML reports... 2022-12-01T10:40:42.9273033Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102647.xml 2022-12-01T10:40:42.9273306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9273430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9273712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9273871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9274058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm5d2_jw 2022-12-01T10:40:42.9274254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm5d2_jw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9274508Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9274542Z 2022-12-01T10:40:42.9274616Z Running tests... 2022-12-01T10:40:42.9274811Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9275032Z test_init_rpc_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24472 2022-12-01T10:40:42.9275191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24473 2022-12-01T10:40:42.9275349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24474 2022-12-01T10:40:42.9275490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24475 2022-12-01T10:40:42.9275764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9275887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9276168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9276305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9276492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdiy6fu8m 2022-12-01T10:40:42.9276689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdiy6fu8m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9276959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9277083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9277349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9277482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9277665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahyg4h5m 2022-12-01T10:40:42.9277863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahyg4h5m/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9278132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9278406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9278685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9278822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9278992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3zz47u4 2022-12-01T10:40:42.9279185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3zz47u4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9279455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9279583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9279862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9279996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9280181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2x9lt4pz 2022-12-01T10:40:42.9280426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2x9lt4pz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9280596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9280749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9280914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9281076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9281293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9281472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9281647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9281821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9282125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9282413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9282704Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9282999Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9283068Z ok (2.340s) 2022-12-01T10:40:42.9283074Z 2022-12-01T10:40:42.9283270Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9283346Z Ran 1 test in 2.340s 2022-12-01T10:40:42.9283351Z 2022-12-01T10:40:42.9283410Z OK 2022-12-01T10:40:42.9283415Z 2022-12-01T10:40:42.9283499Z Generating XML reports... 2022-12-01T10:40:42.9283844Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102650.xml 2022-12-01T10:40:42.9284106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9284228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9284508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9284646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9284831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuugyx1vy 2022-12-01T10:40:42.9285028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuugyx1vy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9285283Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9285288Z 2022-12-01T10:40:42.9285363Z Running tests... 2022-12-01T10:40:42.9285558Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9285762Z test_int_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24871 2022-12-01T10:40:42.9285918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24872 2022-12-01T10:40:42.9286073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24873 2022-12-01T10:40:42.9286228Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24874 2022-12-01T10:40:42.9286501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9286624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9286902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9287065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9287240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv438txy 2022-12-01T10:40:42.9287434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv438txy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9287760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9287926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9288209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9288344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9288530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphz_s0k39 2022-12-01T10:40:42.9288724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphz_s0k39/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9288996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9289105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9289386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9289522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9289708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lkl81ma 2022-12-01T10:40:42.9289904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lkl81ma/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9290175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9290298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9290579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9290717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9290887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdz1eso3 2022-12-01T10:40:42.9291080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdz1eso3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9291248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9291413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9291576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9291739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9291807Z ok (2.142s) 2022-12-01T10:40:42.9291814Z 2022-12-01T10:40:42.9292013Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9292076Z Ran 1 test in 2.142s 2022-12-01T10:40:42.9292080Z 2022-12-01T10:40:42.9292139Z OK 2022-12-01T10:40:42.9292144Z 2022-12-01T10:40:42.9292228Z Generating XML reports... 2022-12-01T10:40:42.9292568Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102654.xml 2022-12-01T10:40:42.9292845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9292967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9293244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9293379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9293598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wkmrwe3 2022-12-01T10:40:42.9293792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wkmrwe3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9294048Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9294053Z 2022-12-01T10:40:42.9294126Z Running tests... 2022-12-01T10:40:42.9294321Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9294566Z test_invalid_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25094 2022-12-01T10:40:42.9294725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25095 2022-12-01T10:40:42.9294881Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25096 2022-12-01T10:40:42.9295035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25097 2022-12-01T10:40:42.9295297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9295420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9295702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9295836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9296024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkf6wr49q 2022-12-01T10:40:42.9296220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkf6wr49q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9296493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9296616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9296899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9297023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9297208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0r2r7yn8 2022-12-01T10:40:42.9297403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0r2r7yn8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9297673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9297799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9298077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9298212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9298397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzh0fpb9 2022-12-01T10:40:42.9298580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzh0fpb9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9298853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9298975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9299256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9299394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9299581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8wcmtvt 2022-12-01T10:40:42.9299778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8wcmtvt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9299944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9300151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9300302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9300464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9300532Z ok (1.839s) 2022-12-01T10:40:42.9300537Z 2022-12-01T10:40:42.9300736Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9300840Z Ran 1 test in 1.839s 2022-12-01T10:40:42.9300845Z 2022-12-01T10:40:42.9300905Z OK 2022-12-01T10:40:42.9300910Z 2022-12-01T10:40:42.9300998Z Generating XML reports... 2022-12-01T10:40:42.9301344Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102658.xml 2022-12-01T10:40:42.9301605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9301736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9302018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9302156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9302343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1e5_0g0 2022-12-01T10:40:42.9302536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1e5_0g0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9302793Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9302798Z 2022-12-01T10:40:42.9302872Z Running tests... 2022-12-01T10:40:42.9303070Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9303283Z test_local_rref_no_fork (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25153 2022-12-01T10:40:42.9303441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25154 2022-12-01T10:40:42.9303598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25155 2022-12-01T10:40:42.9303752Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25156 2022-12-01T10:40:42.9304027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9304153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9304434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9304571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9304743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl732zg2g 2022-12-01T10:40:42.9304942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl732zg2g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9305220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9305345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9305626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9305766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9305949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5h9oyq68 2022-12-01T10:40:42.9306147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5h9oyq68/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9306420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9306557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9306837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9306971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9307157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9aqhukh5 2022-12-01T10:40:42.9307353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9aqhukh5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9307653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9307777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9308054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9308188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9308360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo_idpj9 2022-12-01T10:40:42.9308559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo_idpj9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9308723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9308886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9309050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9309212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9309280Z ok (2.140s) 2022-12-01T10:40:42.9309285Z 2022-12-01T10:40:42.9309482Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9309543Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9309548Z 2022-12-01T10:40:42.9309608Z OK 2022-12-01T10:40:42.9309613Z 2022-12-01T10:40:42.9309703Z Generating XML reports... 2022-12-01T10:40:42.9310042Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102701.xml 2022-12-01T10:40:42.9310316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9310441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9310723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9310858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9311031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46ieofam 2022-12-01T10:40:42.9311226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46ieofam/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9311481Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9311486Z 2022-12-01T10:40:42.9311561Z Running tests... 2022-12-01T10:40:42.9311758Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9311975Z test_local_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25376 2022-12-01T10:40:42.9312131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25377 2022-12-01T10:40:42.9312289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25378 2022-12-01T10:40:42.9312443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25379 2022-12-01T10:40:42.9312704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9312826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9313133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9313268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9313453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvt97rfl 2022-12-01T10:40:42.9313648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvt97rfl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9313919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9314068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9314346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9314466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9314653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtcnbic5 2022-12-01T10:40:42.9314848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtcnbic5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9315117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9315239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9315516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9315653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9315835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ofc1107 2022-12-01T10:40:42.9316013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ofc1107/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9316285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9316406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9316680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9316812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9316992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjricga0g 2022-12-01T10:40:42.9317189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjricga0g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9317353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9317516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9317665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9317830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9318098Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.9318490Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.9318760Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.9319022Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9319333Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9319595Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9319668Z ok (2.040s) 2022-12-01T10:40:42.9319673Z 2022-12-01T10:40:42.9319879Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9319977Z Ran 1 test in 2.040s 2022-12-01T10:40:42.9319995Z 2022-12-01T10:40:42.9320043Z OK 2022-12-01T10:40:42.9320047Z 2022-12-01T10:40:42.9320133Z Generating XML reports... 2022-12-01T10:40:42.9320475Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102705.xml 2022-12-01T10:40:42.9320751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9320878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9321157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9321295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9321483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03ivn553 2022-12-01T10:40:42.9321667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03ivn553/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9321922Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9321927Z 2022-12-01T10:40:42.9322001Z Running tests... 2022-12-01T10:40:42.9322201Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9322428Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25599 2022-12-01T10:40:42.9322585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25600 2022-12-01T10:40:42.9322741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25601 2022-12-01T10:40:42.9322896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25602 2022-12-01T10:40:42.9323169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9323283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9323569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9323706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9323891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsj28cxg 2022-12-01T10:40:42.9324089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsj28cxg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9324361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9324484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9324762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9324887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9325072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlwdvdis 2022-12-01T10:40:42.9325269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlwdvdis/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9325535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9325688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9325970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9326107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9326289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5s0ssaba 2022-12-01T10:40:42.9326483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5s0ssaba/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9326769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9326890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9327169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9327302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9327489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje758tvj 2022-12-01T10:40:42.9327735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje758tvj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9327905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9328070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9328224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9328388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9328567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9328745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9328922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9329228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9329403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9329702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9329999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9330290Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9330545Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9330805Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9331062Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9331326Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:42.9331578Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9331864Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9332120Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9332374Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:42.9332490Z ok (2.139s) 2022-12-01T10:40:42.9332496Z 2022-12-01T10:40:42.9332696Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9332772Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9332777Z 2022-12-01T10:40:42.9332824Z OK 2022-12-01T10:40:42.9332829Z 2022-12-01T10:40:42.9332913Z Generating XML reports... 2022-12-01T10:40:42.9333254Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102708.xml 2022-12-01T10:40:42.9333529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9333654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9333933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9334069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9334260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoo4c3zhu 2022-12-01T10:40:42.9334458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoo4c3zhu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9334701Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9334707Z 2022-12-01T10:40:42.9334780Z Running tests... 2022-12-01T10:40:42.9334980Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9335209Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25834 2022-12-01T10:40:42.9335369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25835 2022-12-01T10:40:42.9335525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25836 2022-12-01T10:40:42.9335679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25837 2022-12-01T10:40:42.9335955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9336064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9336346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9336478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9336665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy82__kq5 2022-12-01T10:40:42.9336857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy82__kq5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9337127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9337249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9337533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9337668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9337837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_03cadp 2022-12-01T10:40:42.9338029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_03cadp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9338339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9338460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9343151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9343292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9343552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_g5bucw 2022-12-01T10:40:42.9343752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_g5bucw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9344034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9344160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9344447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9344585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9344759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1eplm15q 2022-12-01T10:40:42.9344956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1eplm15q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9345122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9345291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9345457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9345621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9345690Z ok (2.039s) 2022-12-01T10:40:42.9345698Z 2022-12-01T10:40:42.9345902Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9345966Z Ran 1 test in 2.040s 2022-12-01T10:40:42.9345983Z 2022-12-01T10:40:42.9346031Z OK 2022-12-01T10:40:42.9346035Z 2022-12-01T10:40:42.9346120Z Generating XML reports... 2022-12-01T10:40:42.9346459Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102712.xml 2022-12-01T10:40:42.9346733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9346859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9347142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9347277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9347460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4k9subd 2022-12-01T10:40:42.9347646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4k9subd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9347901Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9347906Z 2022-12-01T10:40:42.9347979Z Running tests... 2022-12-01T10:40:42.9348178Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9348397Z test_mark_future_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26057 2022-12-01T10:40:42.9348555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26058 2022-12-01T10:40:42.9348711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26059 2022-12-01T10:40:42.9348865Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26060 2022-12-01T10:40:42.9349197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9349311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9349591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9349727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9349912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacawk9eu 2022-12-01T10:40:42.9350135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacawk9eu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9350404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9350525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9350804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9350929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9351110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2je4a7ru 2022-12-01T10:40:42.9351303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2je4a7ru/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9351571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9351698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9351978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9352114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9352299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr4yamj3 2022-12-01T10:40:42.9352494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr4yamj3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9352753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9352875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9353152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9353285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9353468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mth2lsj 2022-12-01T10:40:42.9353664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mth2lsj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9353827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9353992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9354143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9354304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9354373Z ok (2.037s) 2022-12-01T10:40:42.9354378Z 2022-12-01T10:40:42.9354578Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9354654Z Ran 1 test in 2.038s 2022-12-01T10:40:42.9354661Z 2022-12-01T10:40:42.9354720Z OK 2022-12-01T10:40:42.9354725Z 2022-12-01T10:40:42.9354810Z Generating XML reports... 2022-12-01T10:40:42.9355149Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102715.xml 2022-12-01T10:40:42.9355422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9355533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9355837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9355973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9356157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmvrcuhb 2022-12-01T10:40:42.9356353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmvrcuhb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9356640Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9356646Z 2022-12-01T10:40:42.9356721Z Running tests... 2022-12-01T10:40:42.9356919Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9357138Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26280 2022-12-01T10:40:42.9357297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26281 2022-12-01T10:40:42.9357452Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26282 2022-12-01T10:40:42.9357606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26283 2022-12-01T10:40:42.9357884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9358007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9358486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9358624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9358810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgckuktq9 2022-12-01T10:40:42.9358993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgckuktq9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9359272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9359396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9359679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9359815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9360001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mchpep2 2022-12-01T10:40:42.9360197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mchpep2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9360473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9360583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9360865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9361002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9361186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyoqsavz 2022-12-01T10:40:42.9361382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyoqsavz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9361654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9361782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9362063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9362198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9362425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn33o65pc 2022-12-01T10:40:42.9362623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn33o65pc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9362788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9362953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9363117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9363329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9363398Z ok (2.138s) 2022-12-01T10:40:42.9363404Z 2022-12-01T10:40:42.9363603Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9363665Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9363683Z 2022-12-01T10:40:42.9363732Z OK 2022-12-01T10:40:42.9363736Z 2022-12-01T10:40:42.9363821Z Generating XML reports... 2022-12-01T10:40:42.9364166Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102719.xml 2022-12-01T10:40:42.9364439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9364563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9364841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9364980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9365167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo1gj57o 2022-12-01T10:40:42.9365350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo1gj57o/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9365604Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9365609Z 2022-12-01T10:40:42.9365684Z Running tests... 2022-12-01T10:40:42.9365881Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9366118Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26503 2022-12-01T10:40:42.9366275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26504 2022-12-01T10:40:42.9366430Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26505 2022-12-01T10:40:42.9366585Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26506 2022-12-01T10:40:42.9366857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9366968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9367249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9367385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9367570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpab0915fs 2022-12-01T10:40:42.9367830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpab0915fs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9368105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9368232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9368513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9368636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9368823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv48doucm 2022-12-01T10:40:42.9369054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv48doucm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9369326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9369450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9369728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9369892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9370078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4a6pbb4t 2022-12-01T10:40:42.9370273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4a6pbb4t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9370531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9370658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9370937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9371073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9371257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdmru43p 2022-12-01T10:40:42.9371451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdmru43p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9371617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9371781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9371931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9372093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9372164Z ok (2.038s) 2022-12-01T10:40:42.9372169Z 2022-12-01T10:40:42.9372367Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9372444Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9372449Z 2022-12-01T10:40:42.9372510Z OK 2022-12-01T10:40:42.9372514Z 2022-12-01T10:40:42.9372599Z Generating XML reports... 2022-12-01T10:40:42.9372941Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102722.xml 2022-12-01T10:40:42.9373217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9373329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9373609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9373746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9373935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s8sk6vg 2022-12-01T10:40:42.9374131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s8sk6vg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9374385Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9374390Z 2022-12-01T10:40:42.9374465Z Running tests... 2022-12-01T10:40:42.9374663Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9374874Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26726 2022-12-01T10:40:42.9375028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26727 2022-12-01T10:40:42.9375187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26728 2022-12-01T10:40:42.9375367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26729 2022-12-01T10:40:42.9375644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9375767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9376049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9376186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9376404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnivomlo_ 2022-12-01T10:40:42.9376589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnivomlo_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9376861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9376982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9377267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9377402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9377587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpku_03daq 2022-12-01T10:40:42.9377782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpku_03daq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9378058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9378167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9378448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9378581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9378768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoozn8l_b 2022-12-01T10:40:42.9378966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoozn8l_b/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9379235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9379360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9379639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9379778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9379950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9w4rza2 2022-12-01T10:40:42.9380143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9w4rza2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9380312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9380479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9380639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9380800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9380870Z ok (2.138s) 2022-12-01T10:40:42.9380875Z 2022-12-01T10:40:42.9381076Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9381140Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9381158Z 2022-12-01T10:40:42.9381206Z OK 2022-12-01T10:40:42.9381210Z 2022-12-01T10:40:42.9381296Z Generating XML reports... 2022-12-01T10:40:42.9381634Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102726.xml 2022-12-01T10:40:42.9381935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9382060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9382338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9382475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9382662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g6ivfg6 2022-12-01T10:40:42.9382874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g6ivfg6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9383130Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9383135Z 2022-12-01T10:40:42.9383207Z Running tests... 2022-12-01T10:40:42.9383403Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9383617Z test_multi_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26949 2022-12-01T10:40:42.9383773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26950 2022-12-01T10:40:42.9383928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26951 2022-12-01T10:40:42.9384079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26952 2022-12-01T10:40:42.9384353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9384466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9384744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9384880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9385065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczsg_h3p 2022-12-01T10:40:42.9385264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczsg_h3p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9385535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9385657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9385934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9386059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9386243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpscct75tu 2022-12-01T10:40:42.9386486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpscct75tu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9386757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9386881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9387159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9387293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9387478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq57w4p_g 2022-12-01T10:40:42.9387674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq57w4p_g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9387932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9388055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9388332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9388512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9388697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_amdkow 2022-12-01T10:40:42.9388891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_amdkow/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9389055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9389219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9389396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9389557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9389626Z ok (2.140s) 2022-12-01T10:40:42.9389632Z 2022-12-01T10:40:42.9389835Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9389910Z Ran 1 test in 2.141s 2022-12-01T10:40:42.9389915Z 2022-12-01T10:40:42.9389975Z OK 2022-12-01T10:40:42.9389982Z 2022-12-01T10:40:42.9390069Z Generating XML reports... 2022-12-01T10:40:42.9390414Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102729.xml 2022-12-01T10:40:42.9390686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9390799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9391082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9391222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9391405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw_vvksn 2022-12-01T10:40:42.9391600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw_vvksn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9391853Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9391859Z 2022-12-01T10:40:42.9391933Z Running tests... 2022-12-01T10:40:42.9392134Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9392348Z test_my_parameter_server (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27172 2022-12-01T10:40:42.9392503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27173 2022-12-01T10:40:42.9392660Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27174 2022-12-01T10:40:42.9392812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27175 2022-12-01T10:40:42.9393084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9393208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9393490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9393625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9393807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ig4ehl7 2022-12-01T10:40:42.9393988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ig4ehl7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9394260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9394382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9394661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9394794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9395005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputscz44h 2022-12-01T10:40:42.9395202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputscz44h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9395474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9395585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9395863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9396025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9396208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx7n0jz0 2022-12-01T10:40:42.9396402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx7n0jz0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9396679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9396802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9397080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9397213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9397383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9_3lfkh 2022-12-01T10:40:42.9397578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9_3lfkh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9397742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9397906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9398067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9398356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9398427Z ok (2.236s) 2022-12-01T10:40:42.9398432Z 2022-12-01T10:40:42.9398633Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9398695Z Ran 1 test in 2.236s 2022-12-01T10:40:42.9398713Z 2022-12-01T10:40:42.9398760Z OK 2022-12-01T10:40:42.9398765Z 2022-12-01T10:40:42.9398849Z Generating XML reports... 2022-12-01T10:40:42.9399191Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102733.xml 2022-12-01T10:40:42.9399464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9399589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9399872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9400010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9400196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrt_e7qc 2022-12-01T10:40:42.9400378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrt_e7qc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9400634Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9400642Z 2022-12-01T10:40:42.9400715Z Running tests... 2022-12-01T10:40:42.9400914Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9401131Z test_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27431 2022-12-01T10:40:42.9401289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27432 2022-12-01T10:40:42.9401492Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27433 2022-12-01T10:40:42.9401648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27434 2022-12-01T10:40:42.9401911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9402033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9402314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9402491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9402675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xddbcgd 2022-12-01T10:40:42.9402871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xddbcgd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9403144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9403272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9403553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9403674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9403858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qei_k0h 2022-12-01T10:40:42.9404053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qei_k0h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9404327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9404449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9404726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9404860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9405042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uqasl3n 2022-12-01T10:40:42.9405235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uqasl3n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9405493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9405613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9405895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9406029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9406215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7erob21p 2022-12-01T10:40:42.9406411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7erob21p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9406579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9406747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9406895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9407056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9407128Z ok (2.039s) 2022-12-01T10:40:42.9407133Z 2022-12-01T10:40:42.9407333Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9407407Z Ran 1 test in 2.040s 2022-12-01T10:40:42.9407412Z 2022-12-01T10:40:42.9407476Z OK 2022-12-01T10:40:42.9407480Z 2022-12-01T10:40:42.9407565Z Generating XML reports... 2022-12-01T10:40:42.9407959Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102736.xml 2022-12-01T10:40:42.9408278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9408390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9408671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9408807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9409017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nb__yht 2022-12-01T10:40:42.9409208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nb__yht/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9409463Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9409469Z 2022-12-01T10:40:42.9409540Z Running tests... 2022-12-01T10:40:42.9409736Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9409943Z test_nested_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27654 2022-12-01T10:40:42.9410100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27655 2022-12-01T10:40:42.9410254Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27656 2022-12-01T10:40:42.9410409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27657 2022-12-01T10:40:42.9410687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9410811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9411091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9411228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9411416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcmuf4e2 2022-12-01T10:40:42.9411599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcmuf4e2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9411870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9411992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9412274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9412406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9412588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54veud9a 2022-12-01T10:40:42.9412781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54veud9a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9413052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9413161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9413439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9413573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9413756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumgghbq0 2022-12-01T10:40:42.9414027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9414151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9414429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9414562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9414774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc_ozxw4 2022-12-01T10:40:42.9414958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumgghbq0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9415144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc_ozxw4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9415307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9415499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9415661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9415823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9415893Z ok (2.239s) 2022-12-01T10:40:42.9415898Z 2022-12-01T10:40:42.9416096Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9416161Z Ran 1 test in 2.239s 2022-12-01T10:40:42.9416178Z 2022-12-01T10:40:42.9416225Z OK 2022-12-01T10:40:42.9416230Z 2022-12-01T10:40:42.9416317Z Generating XML reports... 2022-12-01T10:40:42.9416657Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102740.xml 2022-12-01T10:40:42.9416930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9417055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9417333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9417467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9417650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n2ju_sb 2022-12-01T10:40:42.9417831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n2ju_sb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9418085Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9418090Z 2022-12-01T10:40:42.9418163Z Running tests... 2022-12-01T10:40:42.9418359Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9418576Z test_nested_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27877 2022-12-01T10:40:42.9418734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27878 2022-12-01T10:40:42.9418892Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27879 2022-12-01T10:40:42.9419045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27880 2022-12-01T10:40:42.9419304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9419430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9419709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9419841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9420026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94uel7_a 2022-12-01T10:40:42.9420223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94uel7_a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9420493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9420614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9420891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9421039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9421223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4b89dvv_ 2022-12-01T10:40:42.9421415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4b89dvv_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9421683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9421832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9422109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9422240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9422423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeq2y6wby 2022-12-01T10:40:42.9422603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeq2y6wby/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9422873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9422997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9423275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9423407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9423591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nphkhpk 2022-12-01T10:40:42.9423786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nphkhpk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9423949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9424112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9424263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9424423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9424490Z ok (2.139s) 2022-12-01T10:40:42.9424494Z 2022-12-01T10:40:42.9424690Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9424766Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9424770Z 2022-12-01T10:40:42.9424833Z OK 2022-12-01T10:40:42.9424837Z 2022-12-01T10:40:42.9424920Z Generating XML reports... 2022-12-01T10:40:42.9425258Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102744.xml 2022-12-01T10:40:42.9425516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9425638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9425921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9426056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9426241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevhws1d9 2022-12-01T10:40:42.9426437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevhws1d9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9426696Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9426701Z 2022-12-01T10:40:42.9426775Z Running tests... 2022-12-01T10:40:42.9426970Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9427179Z test_nested_rref_stress (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28100 2022-12-01T10:40:42.9427338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28101 2022-12-01T10:40:42.9427522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28102 2022-12-01T10:40:42.9427686Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28103 2022-12-01T10:40:42.9427959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9428081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9428389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9428526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9428711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpio1qwnw1 2022-12-01T10:40:42.9428893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpio1qwnw1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9429168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9429293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9429571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9429706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9429891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefze2qqx 2022-12-01T10:40:42.9430091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefze2qqx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9430360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9430468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9430746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9430878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9431062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4ja8vsu 2022-12-01T10:40:42.9431257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4ja8vsu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9431528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9431653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9431933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9432069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9432239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcbthp0k 2022-12-01T10:40:42.9432436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcbthp0k/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9432598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9432764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9432927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9433089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9433156Z ok (2.238s) 2022-12-01T10:40:42.9433161Z 2022-12-01T10:40:42.9433358Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9433420Z Ran 1 test in 2.238s 2022-12-01T10:40:42.9433425Z 2022-12-01T10:40:42.9433484Z OK 2022-12-01T10:40:42.9433488Z 2022-12-01T10:40:42.9433573Z Generating XML reports... 2022-12-01T10:40:42.9433954Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102747.xml 2022-12-01T10:40:42.9434228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9434351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9434631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9434797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9434981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4h23sqh 2022-12-01T10:40:42.9435161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4h23sqh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9435417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9435422Z 2022-12-01T10:40:42.9435495Z Running tests... 2022-12-01T10:40:42.9435693Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9435912Z test_non_cont_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28323 2022-12-01T10:40:42.9436067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28324 2022-12-01T10:40:42.9436223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28325 2022-12-01T10:40:42.9436382Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28326 2022-12-01T10:40:42.9436644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9436764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9437043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9437180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9437364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5e20wu4 2022-12-01T10:40:42.9437559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5e20wu4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9437833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9437960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9438376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9438498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9438682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbwfvq5h 2022-12-01T10:40:42.9438878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbwfvq5h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9439147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9439270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9439549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9439684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9439871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tsvpwln 2022-12-01T10:40:42.9440052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tsvpwln/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9440321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9440443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9440793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9440930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9441114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpug24gxqj 2022-12-01T10:40:42.9441306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpug24gxqj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9441509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9441672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9441821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9441980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9442051Z ok (2.142s) 2022-12-01T10:40:42.9442058Z 2022-12-01T10:40:42.9442262Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9442338Z Ran 1 test in 2.142s 2022-12-01T10:40:42.9442343Z 2022-12-01T10:40:42.9442406Z OK 2022-12-01T10:40:42.9442411Z 2022-12-01T10:40:42.9442497Z Generating XML reports... 2022-12-01T10:40:42.9442840Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102751.xml 2022-12-01T10:40:42.9443103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9443228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9443508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9443644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9443836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtw6v7al 2022-12-01T10:40:42.9444034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtw6v7al/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9444288Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9444293Z 2022-12-01T10:40:42.9444366Z Running tests... 2022-12-01T10:40:42.9444562Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9444822Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28546 2022-12-01T10:40:42.9444981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28547 2022-12-01T10:40:42.9445136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28548 2022-12-01T10:40:42.9445292Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28549 2022-12-01T10:40:42.9445564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9445687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9445965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9446098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9446284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo68kuow 2022-12-01T10:40:42.9446466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo68kuow/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9446740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9446863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9447170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9447307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9447489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dxgkmt0 2022-12-01T10:40:42.9447735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dxgkmt0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9448052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9448163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9448449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9448585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9448772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11nh3lmw 2022-12-01T10:40:42.9448966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11nh3lmw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9449241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9449363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9449643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9449783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9449953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6fwve0z3 2022-12-01T10:40:42.9450149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6fwve0z3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9450317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9450482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9450645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9450807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9450879Z ok (2.138s) 2022-12-01T10:40:42.9450884Z 2022-12-01T10:40:42.9451085Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9451151Z Ran 1 test in 2.138s 2022-12-01T10:40:42.9451168Z 2022-12-01T10:40:42.9451216Z OK 2022-12-01T10:40:42.9451221Z 2022-12-01T10:40:42.9451305Z Generating XML reports... 2022-12-01T10:40:42.9451645Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102754.xml 2022-12-01T10:40:42.9451920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9452044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9452322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9452457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9452642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxay0ih2 2022-12-01T10:40:42.9452827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxay0ih2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9453082Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9453088Z 2022-12-01T10:40:42.9453158Z Running tests... 2022-12-01T10:40:42.9453354Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9453596Z test_nonzero (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28769 2022-12-01T10:40:42.9453752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28770 2022-12-01T10:40:42.9453905Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28771 2022-12-01T10:40:42.9454059Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28772 2022-12-01T10:40:42.9454318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9454469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9454753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9454887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9455069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqm7aqom0 2022-12-01T10:40:42.9455266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqm7aqom0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9455536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9455658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9455935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9456058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9456241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsb2kzg_p 2022-12-01T10:40:42.9456438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsb2kzg_p/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9456708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9456833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9457109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9457242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9457427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstayewyu 2022-12-01T10:40:42.9457607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstayewyu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9457879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9458001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9458277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9458411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9458594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposghfn0q 2022-12-01T10:40:42.9458787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposghfn0q/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9458950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9459112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9459263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9459423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9459491Z ok (2.041s) 2022-12-01T10:40:42.9459496Z 2022-12-01T10:40:42.9459693Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9459772Z Ran 1 test in 2.041s 2022-12-01T10:40:42.9459777Z 2022-12-01T10:40:42.9459837Z OK 2022-12-01T10:40:42.9459868Z 2022-12-01T10:40:42.9459954Z Generating XML reports... 2022-12-01T10:40:42.9460295Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102758.xml 2022-12-01T10:40:42.9460566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9460675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9460984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9461120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9461304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcqyctip 2022-12-01T10:40:42.9461501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcqyctip/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9461756Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9461761Z 2022-12-01T10:40:42.9461833Z Running tests... 2022-12-01T10:40:42.9462030Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9462238Z test_owner_equality (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28992 2022-12-01T10:40:42.9462394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28993 2022-12-01T10:40:42.9462550Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28994 2022-12-01T10:40:42.9462705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28995 2022-12-01T10:40:42.9462982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9463106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9463389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9463523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9463706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkw7q8d81 2022-12-01T10:40:42.9463887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkw7q8d81/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9464162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9464284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9464562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9464694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9464879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3y_o22zi 2022-12-01T10:40:42.9465072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3y_o22zi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9465343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9465453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9465728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9465866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9466050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73bu9dap 2022-12-01T10:40:42.9466246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73bu9dap/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9466549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9466673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9466950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9467083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9467253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64hl1_pv 2022-12-01T10:40:42.9467473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64hl1_pv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9467634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9467799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9467961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9468128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9468198Z ok (2.039s) 2022-12-01T10:40:42.9468203Z 2022-12-01T10:40:42.9468404Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9468467Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9468472Z 2022-12-01T10:40:42.9468536Z OK 2022-12-01T10:40:42.9468540Z 2022-12-01T10:40:42.9468627Z Generating XML reports... 2022-12-01T10:40:42.9468974Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102801.xml 2022-12-01T10:40:42.9469244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9469366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9469645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9469784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9469969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfw60iuq 2022-12-01T10:40:42.9470152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfw60iuq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9470409Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9470416Z 2022-12-01T10:40:42.9470491Z Running tests... 2022-12-01T10:40:42.9470687Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9470914Z test_owner_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29215 2022-12-01T10:40:42.9471074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29216 2022-12-01T10:40:42.9471230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29217 2022-12-01T10:40:42.9471389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29218 2022-12-01T10:40:42.9471648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9471769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9472048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9472185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9472371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpch6m1au5 2022-12-01T10:40:42.9472564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpch6m1au5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9472834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9472989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9473268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9473389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9473569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vswyysb 2022-12-01T10:40:42.9473764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vswyysb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9474082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9474208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9474491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9474629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9474814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6lkkizd 2022-12-01T10:40:42.9474996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6lkkizd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9475272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9475397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9475677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9475811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9475995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswxqi7xd 2022-12-01T10:40:42.9476189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswxqi7xd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9476353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9476515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9476664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9476824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9476896Z ok (2.240s) 2022-12-01T10:40:42.9476901Z 2022-12-01T10:40:42.9477102Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9477177Z Ran 1 test in 2.240s 2022-12-01T10:40:42.9477182Z 2022-12-01T10:40:42.9477243Z OK 2022-12-01T10:40:42.9477247Z 2022-12-01T10:40:42.9477334Z Generating XML reports... 2022-12-01T10:40:42.9477677Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102805.xml 2022-12-01T10:40:42.9477941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9478065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9478500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9478639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9478829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps16e2u09 2022-12-01T10:40:42.9479022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps16e2u09/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9479278Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9479283Z 2022-12-01T10:40:42.9479357Z Running tests... 2022-12-01T10:40:42.9479553Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9479817Z test_pass_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29458 2022-12-01T10:40:42.9479979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29459 2022-12-01T10:40:42.9480136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29460 2022-12-01T10:40:42.9480288Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29461 2022-12-01T10:40:42.9480607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9480730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9481013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9481148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9481335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9e6_okfs 2022-12-01T10:40:42.9481594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9481717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9481994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9482133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9482319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4d31ary 2022-12-01T10:40:42.9482514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9e6_okfs/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9482704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4d31ary/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9482980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9483090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9483367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9483501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9483687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepbe6ayg 2022-12-01T10:40:42.9483884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepbe6ayg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9484152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9484275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9484557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9484691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9484862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6fhvgft_ 2022-12-01T10:40:42.9485055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6fhvgft_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9485218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9485383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9485543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9485703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9485771Z ok (2.139s) 2022-12-01T10:40:42.9485776Z 2022-12-01T10:40:42.9485976Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9486066Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9486072Z 2022-12-01T10:40:42.9486132Z OK 2022-12-01T10:40:42.9486137Z 2022-12-01T10:40:42.9486220Z Generating XML reports... 2022-12-01T10:40:42.9486564Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102809.xml 2022-12-01T10:40:42.9486839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9486993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9487277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9487413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9487597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsv8_ujz 2022-12-01T10:40:42.9487841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsv8_ujz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9488100Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9488106Z 2022-12-01T10:40:42.9488180Z Running tests... 2022-12-01T10:40:42.9488379Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9488603Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29681 2022-12-01T10:40:42.9488762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29682 2022-12-01T10:40:42.9488918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29683 2022-12-01T10:40:42.9489072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29684 2022-12-01T10:40:42.9489333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9489460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9489741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9489877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9490064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsski5bt 2022-12-01T10:40:42.9490265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsski5bt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9490539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9490664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9490942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9491065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9491251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfx7s1x82 2022-12-01T10:40:42.9491445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfx7s1x82/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9491716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9491840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9492119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9492254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9492439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj2cuoie 2022-12-01T10:40:42.9492656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj2cuoie/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9492925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9493045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9493321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9493457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9493672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9g5gx0c 2022-12-01T10:40:42.9493867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9g5gx0c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9494033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9494195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9494346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9494509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9494686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9494862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9495167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9495341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9495645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9495819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9496116Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9496401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9496470Z ok (1.940s) 2022-12-01T10:40:42.9496475Z 2022-12-01T10:40:42.9496672Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9496748Z Ran 1 test in 1.940s 2022-12-01T10:40:42.9496754Z 2022-12-01T10:40:42.9496813Z OK 2022-12-01T10:40:42.9496818Z 2022-12-01T10:40:42.9496903Z Generating XML reports... 2022-12-01T10:40:42.9497249Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102812.xml 2022-12-01T10:40:42.9497525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9497651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9497920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9498057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9498244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1n9k1in 2022-12-01T10:40:42.9498443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1n9k1in/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9498700Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9498705Z 2022-12-01T10:40:42.9498777Z Running tests... 2022-12-01T10:40:42.9498976Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9499193Z test_pickle_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29752 2022-12-01T10:40:42.9499366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29753 2022-12-01T10:40:42.9499526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29754 2022-12-01T10:40:42.9499679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29755 2022-12-01T10:40:42.9499955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9500106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9500388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9500523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9500714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgedix1k 2022-12-01T10:40:42.9500911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgedix1k/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9501169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9501292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9501570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9501706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9501893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdnznjcz 2022-12-01T10:40:42.9502090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdnznjcz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9502361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9502481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9502748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9502886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9503067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp214hc_xr 2022-12-01T10:40:42.9503262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp214hc_xr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9503534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9503656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9503936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9504070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9504257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8s2e1sm 2022-12-01T10:40:42.9504437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8s2e1sm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9504597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9504762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9504928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9505090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9505161Z ok (2.142s) 2022-12-01T10:40:42.9505166Z 2022-12-01T10:40:42.9505366Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9505440Z Ran 1 test in 2.142s 2022-12-01T10:40:42.9505445Z 2022-12-01T10:40:42.9505492Z OK 2022-12-01T10:40:42.9505511Z 2022-12-01T10:40:42.9505614Z Generating XML reports... 2022-12-01T10:40:42.9505956Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102816.xml 2022-12-01T10:40:42.9506229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9506353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9506636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9506800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9506987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyns0p00 2022-12-01T10:40:42.9507184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyns0p00/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9507428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9507433Z 2022-12-01T10:40:42.9507511Z Running tests... 2022-12-01T10:40:42.9507711Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9507937Z test_profiler_export_trace (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29975 2022-12-01T10:40:42.9508093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29976 2022-12-01T10:40:42.9508251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29977 2022-12-01T10:40:42.9508403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29978 2022-12-01T10:40:42.9508677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9508786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9509065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9509201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9509386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5brh67d 2022-12-01T10:40:42.9509581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5brh67d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9509853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9509978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9510256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9510392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9510564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggcn3jq1 2022-12-01T10:40:42.9510758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggcn3jq1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9511027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9511151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9511426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9511563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9511746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf8ivvju 2022-12-01T10:40:42.9511938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf8ivvju/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9512207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9512354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9512633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9512768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9512951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi60aembj 2022-12-01T10:40:42.9513170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi60aembj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9513334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9513499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9513659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9513811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9513878Z ok (2.037s) 2022-12-01T10:40:42.9513883Z 2022-12-01T10:40:42.9514082Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9514156Z Ran 1 test in 2.038s 2022-12-01T10:40:42.9514161Z 2022-12-01T10:40:42.9514221Z OK 2022-12-01T10:40:42.9514226Z 2022-12-01T10:40:42.9514310Z Generating XML reports... 2022-12-01T10:40:42.9514646Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102819.xml 2022-12-01T10:40:42.9514922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9515045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9515310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9515447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9515633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1qg8oj_ 2022-12-01T10:40:42.9515827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1qg8oj_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9516078Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9516083Z 2022-12-01T10:40:42.9516160Z Running tests... 2022-12-01T10:40:42.9516355Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9516592Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30198 2022-12-01T10:40:42.9516736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30199 2022-12-01T10:40:42.9516891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30200 2022-12-01T10:40:42.9517046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30201 2022-12-01T10:40:42.9517320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9517445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9517728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9517865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9518114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5xvkhhp 2022-12-01T10:40:42.9518524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5xvkhhp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9518798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9518986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9519273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9519411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9519597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzz7h2hcc 2022-12-01T10:40:42.9519794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzz7h2hcc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9520108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9520232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9520498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9520633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9520820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp7u0jtm 2022-12-01T10:40:42.9521018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp7u0jtm/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9521287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9521412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9521694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9521829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9522012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd11ukw2 2022-12-01T10:40:42.9522192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd11ukw2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9522361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9522526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9522687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9522851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9522922Z ok (2.037s) 2022-12-01T10:40:42.9522928Z 2022-12-01T10:40:42.9523129Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9523206Z Ran 1 test in 2.037s 2022-12-01T10:40:42.9523211Z 2022-12-01T10:40:42.9523258Z OK 2022-12-01T10:40:42.9523277Z 2022-12-01T10:40:42.9523348Z Generating XML reports... 2022-12-01T10:40:42.9523689Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102823.xml 2022-12-01T10:40:42.9523964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9524090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9524370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9524506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9524694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcq6rz1_ 2022-12-01T10:40:42.9524894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcq6rz1_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9525136Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9525142Z 2022-12-01T10:40:42.9525213Z Running tests... 2022-12-01T10:40:42.9525409Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9525706Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30421 2022-12-01T10:40:42.9525864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30422 2022-12-01T10:40:42.9526018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30423 2022-12-01T10:40:42.9526172Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30424 2022-12-01T10:40:42.9526483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9526606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9526873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9527007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9527195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv0316js 2022-12-01T10:40:42.9527389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv0316js/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9527661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9527840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9528125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9528262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9528432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzln_tnd 2022-12-01T10:40:42.9528628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzln_tnd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9528900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9529024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9529304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9529439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9529624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4hws712 2022-12-01T10:40:42.9529821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4hws712/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9530093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9530204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9530485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9530621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9530805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu65snajt 2022-12-01T10:40:42.9530999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu65snajt/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9531164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9531332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9531493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9531642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9531711Z ok (2.039s) 2022-12-01T10:40:42.9531716Z 2022-12-01T10:40:42.9531914Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9532023Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9532029Z 2022-12-01T10:40:42.9532092Z OK 2022-12-01T10:40:42.9532097Z 2022-12-01T10:40:42.9532184Z Generating XML reports... 2022-12-01T10:40:42.9532526Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102826.xml 2022-12-01T10:40:42.9532799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9532958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9533226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9533362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9533552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1oe0cpk 2022-12-01T10:40:42.9533997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1oe0cpk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9534481Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9534693Z 2022-12-01T10:40:42.9534754Z Running tests... 2022-12-01T10:40:42.9535065Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9535474Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30644 2022-12-01T10:40:42.9535878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30645 2022-12-01T10:40:42.9536203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30646 2022-12-01T10:40:42.9536532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30647 2022-12-01T10:40:42.9536990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9537316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9537755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9538109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9538457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuc3lv8i_ 2022-12-01T10:40:42.9538852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuc3lv8i_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9539345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9539680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9540114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9540453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9540802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwr325j80 2022-12-01T10:40:42.9541201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwr325j80/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9541672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9542008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9542433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9543004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9543338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngodpj_j 2022-12-01T10:40:42.9543793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngodpj_j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9544292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9544629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9545049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9545401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9545875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqhcp3ac 2022-12-01T10:40:42.9546292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqhcp3ac/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9546660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9547014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9547360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9547707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9547949Z ok (2.143s) 2022-12-01T10:40:42.9548050Z 2022-12-01T10:40:42.9548255Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9548498Z Ran 1 test in 2.143s 2022-12-01T10:40:42.9548609Z 2022-12-01T10:40:42.9548658Z OK 2022-12-01T10:40:42.9548750Z 2022-12-01T10:40:42.9548834Z Generating XML reports... 2022-12-01T10:40:42.9549292Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102830.xml 2022-12-01T10:40:42.9549828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9550143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9550575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9550921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9551250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf7s2la3 2022-12-01T10:40:42.9551652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf7s2la3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9552133Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9552342Z 2022-12-01T10:40:42.9552415Z Running tests... 2022-12-01T10:40:42.9552709Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9553108Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30873 2022-12-01T10:40:42.9553522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30874 2022-12-01T10:40:42.9553855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30875 2022-12-01T10:40:42.9554180Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30876 2022-12-01T10:40:42.9554639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9554983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9555428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9555763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9556110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4drs08w 2022-12-01T10:40:42.9556512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4drs08w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9557041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9557361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9557791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9558141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9558600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlzdliah 2022-12-01T10:40:42.9559049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlzdliah/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9559551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9559882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9560305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9560653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9560995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpdr2fy0 2022-12-01T10:40:42.9561454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpdr2fy0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9561956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9562307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9562745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9563091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9563423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8nyp7y3 2022-12-01T10:40:42.9563831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8nyp7y3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9564207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9564556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9564892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9565242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9565500Z ok (2.043s) 2022-12-01T10:40:42.9565606Z 2022-12-01T10:40:42.9565795Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9566037Z Ran 1 test in 2.044s 2022-12-01T10:40:42.9566150Z 2022-12-01T10:40:42.9566211Z OK 2022-12-01T10:40:42.9566303Z 2022-12-01T10:40:42.9566375Z Generating XML reports... 2022-12-01T10:40:42.9566835Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102833.xml 2022-12-01T10:40:42.9567379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9567769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9568217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9568594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9568957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoth5f1e_ 2022-12-01T10:40:42.9569369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoth5f1e_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9569856Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9570068Z 2022-12-01T10:40:42.9570194Z Running tests... 2022-12-01T10:40:42.9570507Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9570915Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31096 2022-12-01T10:40:42.9571346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31097 2022-12-01T10:40:42.9571679Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31098 2022-12-01T10:40:42.9572047Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31099 2022-12-01T10:40:42.9572517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9572843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9573276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9573630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9573985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5koe39w 2022-12-01T10:40:42.9574382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5koe39w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9574876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9575215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9575637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9575990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9576346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwr_l_tqv 2022-12-01T10:40:42.9576758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwr_l_tqv/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9577234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9577569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9578008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9578367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9578710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mh9_oht 2022-12-01T10:40:42.9579119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mh9_oht/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9579611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9579932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9580375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9580726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9581072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f6jyc5w 2022-12-01T10:40:42.9581464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f6jyc5w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9581849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9582210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9582562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9582905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9583154Z ok (2.040s) 2022-12-01T10:40:42.9583321Z 2022-12-01T10:40:42.9583524Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9583756Z Ran 1 test in 2.040s 2022-12-01T10:40:42.9583871Z 2022-12-01T10:40:42.9583935Z OK 2022-12-01T10:40:42.9584026Z 2022-12-01T10:40:42.9584111Z Generating XML reports... 2022-12-01T10:40:42.9584564Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102837.xml 2022-12-01T10:40:42.9585358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9585711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9586148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9586486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9586843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdiph0ohh 2022-12-01T10:40:42.9587257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdiph0ohh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9587737Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9587934Z 2022-12-01T10:40:42.9588009Z Running tests... 2022-12-01T10:40:42.9588323Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9588730Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31319 2022-12-01T10:40:42.9589125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31320 2022-12-01T10:40:42.9589461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31321 2022-12-01T10:40:42.9589795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31322 2022-12-01T10:40:42.9590261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9590589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9591023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9591382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9591719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2ei1heb 2022-12-01T10:40:42.9592125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2ei1heb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9592617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9592947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9593371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9593718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9594066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgy1tgmdh 2022-12-01T10:40:42.9594554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9594876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9595305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9595653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9595984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi09iis5i 2022-12-01T10:40:42.9596470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgy1tgmdh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9596887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi09iis5i/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9597376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9597703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9598142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9598763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9599170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz0y3nh4 2022-12-01T10:40:42.9599610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz0y3nh4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9600028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9600387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9600722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9601065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9601320Z ok (2.138s) 2022-12-01T10:40:42.9601426Z 2022-12-01T10:40:42.9601639Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9601874Z Ran 1 test in 2.138s 2022-12-01T10:40:42.9601992Z 2022-12-01T10:40:42.9602054Z OK 2022-12-01T10:40:42.9602144Z 2022-12-01T10:40:42.9602230Z Generating XML reports... 2022-12-01T10:40:42.9602684Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102840.xml 2022-12-01T10:40:42.9603232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9603571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9603999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9604335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9604684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgz5he15 2022-12-01T10:40:42.9605090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgz5he15/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9605545Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9605756Z 2022-12-01T10:40:42.9605832Z Running tests... 2022-12-01T10:40:42.9606191Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9606614Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31542 2022-12-01T10:40:42.9607027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31543 2022-12-01T10:40:42.9607358Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31544 2022-12-01T10:40:42.9607756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31545 2022-12-01T10:40:42.9608206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9608549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9608985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9609335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9609668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbk7male 2022-12-01T10:40:42.9610660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbk7male/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9611273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9611617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9612034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9612439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9612792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphr15_lag 2022-12-01T10:40:42.9613182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphr15_lag/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9613673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9614013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9614450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9614782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9615127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ru5j3y7 2022-12-01T10:40:42.9615531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ru5j3y7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9616006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9616338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9616768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9617120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9617454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwwfntts 2022-12-01T10:40:42.9617860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwwfntts/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9618239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9618591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9618931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9619274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9619526Z ok (2.138s) 2022-12-01T10:40:42.9619631Z 2022-12-01T10:40:42.9619823Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9620069Z Ran 1 test in 2.138s 2022-12-01T10:40:42.9620181Z 2022-12-01T10:40:42.9620245Z OK 2022-12-01T10:40:42.9620338Z 2022-12-01T10:40:42.9620425Z Generating XML reports... 2022-12-01T10:40:42.9620875Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102844.xml 2022-12-01T10:40:42.9621416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9621886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9622313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9622671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9623021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06l2pe45 2022-12-01T10:40:42.9623428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06l2pe45/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9623935Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9624148Z 2022-12-01T10:40:42.9624223Z Running tests... 2022-12-01T10:40:42.9624531Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9624919Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31765 2022-12-01T10:40:42.9625353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31766 2022-12-01T10:40:42.9625725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31767 2022-12-01T10:40:42.9626061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31768 2022-12-01T10:40:42.9626507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9626839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9627270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9627616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9627953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvq7_c7n 2022-12-01T10:40:42.9628356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvq7_c7n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9628847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9629164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9629596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9629943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9630289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1big1s2d 2022-12-01T10:40:42.9630679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1big1s2d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9631175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9631509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9631940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9632288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9632633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyplwny6e 2022-12-01T10:40:42.9633035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyplwny6e/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9633877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9634286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9634725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9635091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9635439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fivsx9e 2022-12-01T10:40:42.9635861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fivsx9e/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9636249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9636588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9636993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9637341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9637795Z STAGE:2022-12-01 10:28:51 31766:31766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-12-01T10:40:42.9638334Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator()) 2022-12-01T10:40:42.9638843Z STAGE:2022-12-01 10:28:52 31766:31766 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-12-01T10:40:42.9639175Z ok (5.143s) 2022-12-01T10:40:42.9639283Z 2022-12-01T10:40:42.9639474Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9639725Z Ran 1 test in 5.144s 2022-12-01T10:40:42.9639839Z 2022-12-01T10:40:42.9639902Z OK 2022-12-01T10:40:42.9639994Z 2022-12-01T10:40:42.9640080Z Generating XML reports... 2022-12-01T10:40:42.9640528Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102847.xml 2022-12-01T10:40:42.9641071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9641403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9641822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9642173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9642523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgl4opphc 2022-12-01T10:40:42.9642932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgl4opphc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9643390Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9643599Z 2022-12-01T10:40:42.9643672Z Running tests... 2022-12-01T10:40:42.9643981Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9644397Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31989 2022-12-01T10:40:42.9644798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31990 2022-12-01T10:40:42.9645133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31991 2022-12-01T10:40:42.9645465Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31992 2022-12-01T10:40:42.9645906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9646238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9646664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9647016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9647347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunl5fz6g 2022-12-01T10:40:42.9647811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunl5fz6g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9648304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9648641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9649061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9649412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9649779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpys38r9ra 2022-12-01T10:40:42.9650408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpys38r9ra/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9650908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9651237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9651666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9651998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9652383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvhnucuo 2022-12-01T10:40:42.9652788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvhnucuo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9653259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9653590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9654018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9654360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9654689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxnfhj1h 2022-12-01T10:40:42.9655093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxnfhj1h/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9655475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9655823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9656158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9656500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9656944Z STAGE:2022-12-01 10:28:58 31990:31990 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-12-01T10:40:42.9657338Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator()) 2022-12-01T10:40:42.9657823Z STAGE:2022-12-01 10:28:59 31990:31990 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-12-01T10:40:42.9658078Z ok (5.152s) 2022-12-01T10:40:42.9658183Z 2022-12-01T10:40:42.9658387Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9658616Z Ran 1 test in 5.152s 2022-12-01T10:40:42.9658726Z 2022-12-01T10:40:42.9658787Z OK 2022-12-01T10:40:42.9658878Z 2022-12-01T10:40:42.9658964Z Generating XML reports... 2022-12-01T10:40:42.9659411Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102854.xml 2022-12-01T10:40:42.9659952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9660288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9660718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9661119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9661475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgmmsz5y 2022-12-01T10:40:42.9661884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgmmsz5y/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9662349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9662558Z 2022-12-01T10:40:42.9662633Z Running tests... 2022-12-01T10:40:42.9662934Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9663372Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32213 2022-12-01T10:40:42.9663767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32214 2022-12-01T10:40:42.9664102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32215 2022-12-01T10:40:42.9664433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32216 2022-12-01T10:40:42.9664884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9665239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9665669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9666015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9666347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqecasea 2022-12-01T10:40:42.9666753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqecasea/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9667243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9667571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9667990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9668342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9668689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kihkgg2 2022-12-01T10:40:42.9669077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kihkgg2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9669564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9669897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9670328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9670660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9671002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1t7num1b 2022-12-01T10:40:42.9671406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1t7num1b/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9671895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9672219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9672654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9673001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9673334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi614legd 2022-12-01T10:40:42.9673735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi614legd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9674117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9674469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9674809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9675151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9675402Z ok (2.139s) 2022-12-01T10:40:42.9675505Z 2022-12-01T10:40:42.9675693Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9675937Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9676052Z 2022-12-01T10:40:42.9676171Z OK 2022-12-01T10:40:42.9676264Z 2022-12-01T10:40:42.9676350Z Generating XML reports... 2022-12-01T10:40:42.9676795Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102900.xml 2022-12-01T10:40:42.9677334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9677666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9678121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9678584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9678932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4xj4cra 2022-12-01T10:40:42.9679339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4xj4cra/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9679798Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9680009Z 2022-12-01T10:40:42.9680085Z Running tests... 2022-12-01T10:40:42.9680397Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9680819Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32436 2022-12-01T10:40:42.9681226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32437 2022-12-01T10:40:42.9681561Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32438 2022-12-01T10:40:42.9681895Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32439 2022-12-01T10:40:42.9682337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9682676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9683107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9683457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9683788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hshq11u 2022-12-01T10:40:42.9684193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hshq11u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9684686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9685002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9685431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9685775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9686122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzg1nfm8r 2022-12-01T10:40:42.9686513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzg1nfm8r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9687001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9687331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9687985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9688329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9688676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhkkz6h4 2022-12-01T10:40:42.9689082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhkkz6h4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9689643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9689975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9690408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9690758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9691138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6ax9n1w 2022-12-01T10:40:42.9691540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6ax9n1w/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9691918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9692253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9692601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9692945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9693192Z ok (2.037s) 2022-12-01T10:40:42.9693296Z 2022-12-01T10:40:42.9693485Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9693730Z Ran 1 test in 2.037s 2022-12-01T10:40:42.9693843Z 2022-12-01T10:40:42.9693911Z OK 2022-12-01T10:40:42.9694006Z 2022-12-01T10:40:42.9694076Z Generating XML reports... 2022-12-01T10:40:42.9694532Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102904.xml 2022-12-01T10:40:42.9695070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9695399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9695817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9696163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9696512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkp0kit3 2022-12-01T10:40:42.9696920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkp0kit3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9697380Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9697591Z 2022-12-01T10:40:42.9697664Z Running tests... 2022-12-01T10:40:42.9697964Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9698353Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32659 2022-12-01T10:40:42.9698749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32660 2022-12-01T10:40:42.9699087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32661 2022-12-01T10:40:42.9699419Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32662 2022-12-01T10:40:42.9699860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9700191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9700621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9700956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9701304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixmrjqhr 2022-12-01T10:40:42.9701876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixmrjqhr/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9702444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9702765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9703195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9703541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9703889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnz3kz7mj 2022-12-01T10:40:42.9704311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnz3kz7mj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9704800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9705132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9705552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9705898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9706244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhxs6v9t 2022-12-01T10:40:42.9706648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhxs6v9t/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9707120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9707455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9707889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9708234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9708568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrlthp41 2022-12-01T10:40:42.9708971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrlthp41/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9709345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9709682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9710030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9710376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9710620Z ok (2.145s) 2022-12-01T10:40:42.9710711Z 2022-12-01T10:40:42.9710911Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9711155Z Ran 1 test in 2.145s 2022-12-01T10:40:42.9711268Z 2022-12-01T10:40:42.9711328Z OK 2022-12-01T10:40:42.9711419Z 2022-12-01T10:40:42.9711493Z Generating XML reports... 2022-12-01T10:40:42.9711951Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102907.xml 2022-12-01T10:40:42.9712488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9712818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9713236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9713585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9713931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnub0vwgg 2022-12-01T10:40:42.9714324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnub0vwgg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9714797Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9715007Z 2022-12-01T10:40:42.9715082Z Running tests... 2022-12-01T10:40:42.9715431Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9715835Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 414 2022-12-01T10:40:42.9716258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 415 2022-12-01T10:40:42.9716575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 416 2022-12-01T10:40:42.9716929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 417 2022-12-01T10:40:42.9717379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9717710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9718127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9718698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9719057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg36fcgyu 2022-12-01T10:40:42.9719454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg36fcgyu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9719950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9720287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9720722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9721059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9721406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhesrngq 2022-12-01T10:40:42.9721811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhesrngq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9722289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9722629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9723063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9723411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9723743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz50iijl8 2022-12-01T10:40:42.9723941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz50iijl8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9724215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9724341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9724627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9724767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9724955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6ogmmz8 2022-12-01T10:40:42.9725152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6ogmmz8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9725306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9725475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9725641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9725805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9725874Z ok (2.139s) 2022-12-01T10:40:42.9725879Z 2022-12-01T10:40:42.9726132Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9726213Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9726218Z 2022-12-01T10:40:42.9726281Z OK 2022-12-01T10:40:42.9726286Z 2022-12-01T10:40:42.9726372Z Generating XML reports... 2022-12-01T10:40:42.9726699Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102911.xml 2022-12-01T10:40:42.9727018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9727143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9727425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9727559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9727806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4p4cjmb 2022-12-01T10:40:42.9728007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4p4cjmb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9728266Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9728272Z 2022-12-01T10:40:42.9728349Z Running tests... 2022-12-01T10:40:42.9728542Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9728774Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 637 2022-12-01T10:40:42.9728932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 638 2022-12-01T10:40:42.9729086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 639 2022-12-01T10:40:42.9729241Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 640 2022-12-01T10:40:42.9729516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9729640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9729926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9730048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9730234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk72l7is_ 2022-12-01T10:40:42.9730432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk72l7is_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9730707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9730834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9731117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9731249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9731436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjd9pkav 2022-12-01T10:40:42.9731629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjd9pkav/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9731888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9732012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9732291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9732421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9732603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcw60nyr2 2022-12-01T10:40:42.9732837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcw60nyr2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9733115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9733239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9733519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9733669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9733857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0je8a18 2022-12-01T10:40:42.9734057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0je8a18/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9734218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9734383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9734544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9734705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9734770Z ok (5.145s) 2022-12-01T10:40:42.9734776Z 2022-12-01T10:40:42.9734964Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9735045Z Ran 1 test in 5.145s 2022-12-01T10:40:42.9735049Z 2022-12-01T10:40:42.9735110Z OK 2022-12-01T10:40:42.9735115Z 2022-12-01T10:40:42.9735199Z Generating XML reports... 2022-12-01T10:40:42.9735543Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102915.xml 2022-12-01T10:40:42.9735817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9735945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9736225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9736358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9736528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mq5u9yk 2022-12-01T10:40:42.9736720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mq5u9yk/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9736978Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9736983Z 2022-12-01T10:40:42.9737055Z Running tests... 2022-12-01T10:40:42.9737254Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9737500Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 862 2022-12-01T10:40:42.9737658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 863 2022-12-01T10:40:42.9737804Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 864 2022-12-01T10:40:42.9737944Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 865 2022-12-01T10:40:42.9738218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9738346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9738626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9738757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9738944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yzieanp 2022-12-01T10:40:42.9739194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yzieanp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9739469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9739590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9739856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9740018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9740200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zb3a3jo 2022-12-01T10:40:42.9740394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zb3a3jo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9740662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9740789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9741067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9741199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9741384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocwu26gd 2022-12-01T10:40:42.9741563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocwu26gd/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9741837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9741958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9742237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9742373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9742560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp0zcb_a 2022-12-01T10:40:42.9742757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp0zcb_a/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9742919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9743086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9743238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9743399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9743465Z ok (5.144s) 2022-12-01T10:40:42.9743470Z 2022-12-01T10:40:42.9743674Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9743749Z Ran 1 test in 5.144s 2022-12-01T10:40:42.9743754Z 2022-12-01T10:40:42.9743814Z OK 2022-12-01T10:40:42.9743819Z 2022-12-01T10:40:42.9743906Z Generating XML reports... 2022-12-01T10:40:42.9744249Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102921.xml 2022-12-01T10:40:42.9744509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9744635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9744918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9745052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9745241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvo1bgcx 2022-12-01T10:40:42.9745434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvo1bgcx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9745720Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9745726Z 2022-12-01T10:40:42.9745799Z Running tests... 2022-12-01T10:40:42.9746000Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9746222Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1085 2022-12-01T10:40:42.9746379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1086 2022-12-01T10:40:42.9746563Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1087 2022-12-01T10:40:42.9746714Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1088 2022-12-01T10:40:42.9746994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9747119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9747401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9747539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9747725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekngvte2 2022-12-01T10:40:42.9747909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekngvte2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9748184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9748306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9748587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9748720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9748909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk_ixtm_ 2022-12-01T10:40:42.9749100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk_ixtm_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9749372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9749481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9749756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9749894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9750077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1luyaf7 2022-12-01T10:40:42.9750271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1luyaf7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9750543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9750668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9750943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9751074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9751246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmeplcvk8 2022-12-01T10:40:42.9751444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmeplcvk8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9751612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9751777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9751934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9752128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9752195Z ok (2.139s) 2022-12-01T10:40:42.9752200Z 2022-12-01T10:40:42.9752400Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9752463Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9752480Z 2022-12-01T10:40:42.9752530Z OK 2022-12-01T10:40:42.9752535Z 2022-12-01T10:40:42.9752617Z Generating XML reports... 2022-12-01T10:40:42.9752960Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102928.xml 2022-12-01T10:40:42.9753266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9753387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9753668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9753803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9753984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4u0nodff 2022-12-01T10:40:42.9754166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4u0nodff/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9754422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9754428Z 2022-12-01T10:40:42.9754505Z Running tests... 2022-12-01T10:40:42.9754702Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9754955Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1308 2022-12-01T10:40:42.9755110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1309 2022-12-01T10:40:42.9755262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1310 2022-12-01T10:40:42.9755418Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1311 2022-12-01T10:40:42.9755693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9755803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9756083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9756224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9756409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jljqepw 2022-12-01T10:40:42.9756606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jljqepw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9756885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9757008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9757291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9757424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9757596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpanjzu2cq 2022-12-01T10:40:42.9757788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpanjzu2cq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9758063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9758312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9758596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9758797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9759140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgh7jfash 2022-12-01T10:40:42.9759341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgh7jfash/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9759619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9759729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9760056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9760194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9760380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsx3b9iy 2022-12-01T10:40:42.9760577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsx3b9iy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9760745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9760918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9761082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9761248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9761308Z ok (2.041s) 2022-12-01T10:40:42.9761313Z 2022-12-01T10:40:42.9761515Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9761593Z Ran 1 test in 2.041s 2022-12-01T10:40:42.9761598Z 2022-12-01T10:40:42.9761663Z OK 2022-12-01T10:40:42.9761667Z 2022-12-01T10:40:42.9761753Z Generating XML reports... 2022-12-01T10:40:42.9762096Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102931.xml 2022-12-01T10:40:42.9762375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9762500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9762766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9762905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9763093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnty55xi 2022-12-01T10:40:42.9763294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnty55xi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9763555Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9763561Z 2022-12-01T10:40:42.9763638Z Running tests... 2022-12-01T10:40:42.9763840Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9764081Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1531 2022-12-01T10:40:42.9764240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1532 2022-12-01T10:40:42.9764380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1533 2022-12-01T10:40:42.9764533Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1534 2022-12-01T10:40:42.9764812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9764939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9765221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9765359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9765579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_hrx4so 2022-12-01T10:40:42.9765780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_hrx4so/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9766053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9766163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9766480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9766613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9766803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal9xz5z_ 2022-12-01T10:40:42.9766999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal9xz5z_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9767276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9767405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9767772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9767898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9768082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ccc936r 2022-12-01T10:40:42.9768280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ccc936r/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9768551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9768672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9768956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9769090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9769273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpph75mitg 2022-12-01T10:40:42.9769472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpph75mitg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9769623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9769793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9769953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9770116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9770181Z ok (2.238s) 2022-12-01T10:40:42.9770186Z 2022-12-01T10:40:42.9770386Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9770464Z Ran 1 test in 2.238s 2022-12-01T10:40:42.9770469Z 2022-12-01T10:40:42.9770528Z OK 2022-12-01T10:40:42.9770533Z 2022-12-01T10:40:42.9770615Z Generating XML reports... 2022-12-01T10:40:42.9770943Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102935.xml 2022-12-01T10:40:42.9771216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9771340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9771623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9771756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9771937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ohza9tn 2022-12-01T10:40:42.9772169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ohza9tn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9772426Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9772431Z 2022-12-01T10:40:42.9772490Z Running tests... 2022-12-01T10:40:42.9772683Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9772939Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1754 2022-12-01T10:40:42.9773119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1755 2022-12-01T10:40:42.9773271Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1756 2022-12-01T10:40:42.9773424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1757 2022-12-01T10:40:42.9773699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9773825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9774105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9774229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9774414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ppm60fx 2022-12-01T10:40:42.9774614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ppm60fx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9774885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9775006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9775284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9775418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9775599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivbiulnh 2022-12-01T10:40:42.9775797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivbiulnh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9776056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9776186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9776466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9776601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9776783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkp3_skw 2022-12-01T10:40:42.9776982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkp3_skw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9777251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9777373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9777638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9777778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9777961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmuo9oaf 2022-12-01T10:40:42.9778155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmuo9oaf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9778319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9778480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9778675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9778839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9778904Z ok (2.142s) 2022-12-01T10:40:42.9778909Z 2022-12-01T10:40:42.9779094Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9779167Z Ran 1 test in 2.142s 2022-12-01T10:40:42.9779199Z 2022-12-01T10:40:42.9779263Z OK 2022-12-01T10:40:42.9779267Z 2022-12-01T10:40:42.9779350Z Generating XML reports... 2022-12-01T10:40:42.9779692Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102939.xml 2022-12-01T10:40:42.9779970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9780093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9780378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9780511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9780684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcley6of 2022-12-01T10:40:42.9780876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcley6of/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9781134Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9781139Z 2022-12-01T10:40:42.9781208Z Running tests... 2022-12-01T10:40:42.9781402Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9781638Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1977 2022-12-01T10:40:42.9781798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1978 2022-12-01T10:40:42.9781953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1979 2022-12-01T10:40:42.9782092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1980 2022-12-01T10:40:42.9782369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9782490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9782775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9782908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9783092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9owpguq6 2022-12-01T10:40:42.9783289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9owpguq6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9783564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9783686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9783952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9784089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9784275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphew6vtch 2022-12-01T10:40:42.9784471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphew6vtch/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9784740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9784860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9785171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9785307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9785478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisy6q904 2022-12-01T10:40:42.9785671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisy6q904/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9785972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9786094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9796458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9796679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9796871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tqnv_eh 2022-12-01T10:40:42.9797076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tqnv_eh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9797246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9797415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9797579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9797747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9797818Z ok (2.138s) 2022-12-01T10:40:42.9797826Z 2022-12-01T10:40:42.9798080Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9798144Z Ran 1 test in 2.138s 2022-12-01T10:40:42.9798165Z 2022-12-01T10:40:42.9798349Z OK 2022-12-01T10:40:42.9798354Z 2022-12-01T10:40:42.9798442Z Generating XML reports... 2022-12-01T10:40:42.9798803Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102942.xml 2022-12-01T10:40:42.9799083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9799210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9799495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9799636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9799825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphh9xp36c 2022-12-01T10:40:42.9800010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphh9xp36c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9800271Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9800279Z 2022-12-01T10:40:42.9800353Z Running tests... 2022-12-01T10:40:42.9800555Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9800804Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2200 2022-12-01T10:40:42.9800964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2201 2022-12-01T10:40:42.9801121Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2202 2022-12-01T10:40:42.9801274Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2203 2022-12-01T10:40:42.9801552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9801664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9802074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9802215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9802406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g54oc8b 2022-12-01T10:40:42.9802603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g54oc8b/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9802881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9803051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9803337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9803477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9803648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo3b_2jg 2022-12-01T10:40:42.9803848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo3b_2jg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9804120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9804243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9804524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9804664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9804848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fk1btb6 2022-12-01T10:40:42.9805044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fk1btb6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9805318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9805429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9805712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9805846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9806034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjzrjdpb 2022-12-01T10:40:42.9806233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjzrjdpb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9806404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9806569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9806732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9806882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9806952Z ok (2.139s) 2022-12-01T10:40:42.9806957Z 2022-12-01T10:40:42.9807156Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9807233Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9807238Z 2022-12-01T10:40:42.9807302Z OK 2022-12-01T10:40:42.9807306Z 2022-12-01T10:40:42.9807392Z Generating XML reports... 2022-12-01T10:40:42.9807804Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102946.xml 2022-12-01T10:40:42.9808085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9808213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9808482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9808620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9808846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcxb3_5b 2022-12-01T10:40:42.9809044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcxb3_5b/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9809301Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9809306Z 2022-12-01T10:40:42.9809379Z Running tests... 2022-12-01T10:40:42.9809607Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9809844Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2423 2022-12-01T10:40:42.9809988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2424 2022-12-01T10:40:42.9810142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2425 2022-12-01T10:40:42.9810300Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2426 2022-12-01T10:40:42.9810576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9810703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9810983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9811121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9811309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ah_n29c 2022-12-01T10:40:42.9811508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ah_n29c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9811766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9811890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9812176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9812312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9812496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7812ntaw 2022-12-01T10:40:42.9812693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7812ntaw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9812967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9813093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9813373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9813495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9813683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcnf0qov 2022-12-01T10:40:42.9813879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcnf0qov/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9814149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9814275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9814561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9814696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9814881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50bwy7_d 2022-12-01T10:40:42.9815062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50bwy7_d/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9815270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9815436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9815600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9815762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9815829Z ok (2.038s) 2022-12-01T10:40:42.9815862Z 2022-12-01T10:40:42.9816063Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9816139Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9816144Z 2022-12-01T10:40:42.9816210Z OK 2022-12-01T10:40:42.9816215Z 2022-12-01T10:40:42.9816286Z Generating XML reports... 2022-12-01T10:40:42.9816628Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102949.xml 2022-12-01T10:40:42.9816905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9817030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9817312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9817448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9817637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldl6ylvb 2022-12-01T10:40:42.9817838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldl6ylvb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9818094Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9818100Z 2022-12-01T10:40:42.9818159Z Running tests... 2022-12-01T10:40:42.9818357Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9818611Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2646 2022-12-01T10:40:42.9818768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2647 2022-12-01T10:40:42.9818921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2648 2022-12-01T10:40:42.9819073Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2649 2022-12-01T10:40:42.9819352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9819476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9819759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9819880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9820069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgydb6707 2022-12-01T10:40:42.9820266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgydb6707/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9820541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9820664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9820945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9821082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9821270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtomugn7 2022-12-01T10:40:42.9821452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtomugn7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9821750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9821877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9822159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9822294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9822476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynl9v_83 2022-12-01T10:40:42.9822701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynl9v_83/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9822974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9823098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9823362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9823505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9823688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvvtm5cw 2022-12-01T10:40:42.9823882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvvtm5cw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9824050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9824212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9824376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9824539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9824608Z ok (2.139s) 2022-12-01T10:40:42.9824613Z 2022-12-01T10:40:42.9824799Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9824876Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9824884Z 2022-12-01T10:40:42.9824945Z OK 2022-12-01T10:40:42.9824950Z 2022-12-01T10:40:42.9825037Z Generating XML reports... 2022-12-01T10:40:42.9825378Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102953.xml 2022-12-01T10:40:42.9825651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9825777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9826058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9826180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9826363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ida9mwe 2022-12-01T10:40:42.9826561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ida9mwe/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9826814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9826819Z 2022-12-01T10:40:42.9826893Z Running tests... 2022-12-01T10:40:42.9827089Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9827321Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2869 2022-12-01T10:40:42.9827480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2870 2022-12-01T10:40:42.9827634Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2871 2022-12-01T10:40:42.9827772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2872 2022-12-01T10:40:42.9828044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9828201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9828485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9828620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9828808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12t2_1vz 2022-12-01T10:40:42.9829001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12t2_1vz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9829303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9829427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9829693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9829834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9830017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdc957eje 2022-12-01T10:40:42.9830213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdc957eje/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9830488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9830614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9830899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9831033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9831205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwoehu5m1 2022-12-01T10:40:42.9831400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwoehu5m1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9831673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9831797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9832076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9832211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9832398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd3xgu49 2022-12-01T10:40:42.9832590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd3xgu49/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9832758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9832909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9833073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9833235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9833303Z ok (4.147s) 2022-12-01T10:40:42.9833308Z 2022-12-01T10:40:42.9833507Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9833583Z Ran 1 test in 4.147s 2022-12-01T10:40:42.9833588Z 2022-12-01T10:40:42.9833650Z OK 2022-12-01T10:40:42.9833656Z 2022-12-01T10:40:42.9833740Z Generating XML reports... 2022-12-01T10:40:42.9834064Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102956.xml 2022-12-01T10:40:42.9834340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9834464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9834772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9834908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9835094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjemhczz 2022-12-01T10:40:42.9835290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjemhczz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9835549Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9835594Z 2022-12-01T10:40:42.9835667Z Running tests... 2022-12-01T10:40:42.9835851Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9836097Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3092 2022-12-01T10:40:42.9836254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3093 2022-12-01T10:40:42.9836409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3094 2022-12-01T10:40:42.9836563Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3095 2022-12-01T10:40:42.9836838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9836966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9837249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9837385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9837557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fno6hnh 2022-12-01T10:40:42.9837752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fno6hnh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9838025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9838150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9838558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9838695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9838880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkqg2rvn 2022-12-01T10:40:42.9839079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkqg2rvn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9839336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9839461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9839742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9839878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9840063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05nw40xf 2022-12-01T10:40:42.9840254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05nw40xf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9840525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9840652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9840931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9841052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9841235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zqlvse0 2022-12-01T10:40:42.9841478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zqlvse0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9841644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9841809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9841972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9842167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9842234Z ok (4.142s) 2022-12-01T10:40:42.9842240Z 2022-12-01T10:40:42.9842426Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9842502Z Ran 1 test in 4.142s 2022-12-01T10:40:42.9842507Z 2022-12-01T10:40:42.9842569Z OK 2022-12-01T10:40:42.9842574Z 2022-12-01T10:40:42.9842658Z Generating XML reports... 2022-12-01T10:40:42.9843003Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103002.xml 2022-12-01T10:40:42.9843282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9843406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9843685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9843823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9843997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogs5kgf6 2022-12-01T10:40:42.9844200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogs5kgf6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9844457Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9844462Z 2022-12-01T10:40:42.9844535Z Running tests... 2022-12-01T10:40:42.9844738Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9844955Z test_py_built_in (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3315 2022-12-01T10:40:42.9845136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3316 2022-12-01T10:40:42.9845356Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3317 2022-12-01T10:40:42.9845510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3318 2022-12-01T10:40:42.9845790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9845916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9846195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9846336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9846522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3824hpgq 2022-12-01T10:40:42.9846720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3824hpgq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9846993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9847118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9847384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9847519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9847755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwovvknvf 2022-12-01T10:40:42.9847995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwovvknvf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9848267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9848392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9848669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9848805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9849040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r_8dni2 2022-12-01T10:40:42.9849217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r_8dni2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9849490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9849613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9849889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9850021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9850208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwp6ohy13 2022-12-01T10:40:42.9850400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwp6ohy13/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9850564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9850715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9850877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9851037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9851105Z ok (2.137s) 2022-12-01T10:40:42.9851111Z 2022-12-01T10:40:42.9851312Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9851389Z Ran 1 test in 2.138s 2022-12-01T10:40:42.9851394Z 2022-12-01T10:40:42.9851457Z OK 2022-12-01T10:40:42.9851462Z 2022-12-01T10:40:42.9851546Z Generating XML reports... 2022-12-01T10:40:42.9851883Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103008.xml 2022-12-01T10:40:42.9852145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9852267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9852546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9852680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9852867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1t1xrsp 2022-12-01T10:40:42.9853062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1t1xrsp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9853317Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9853322Z 2022-12-01T10:40:42.9853393Z Running tests... 2022-12-01T10:40:42.9853577Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9853799Z test_py_class_constructor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3538 2022-12-01T10:40:42.9853955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3539 2022-12-01T10:40:42.9854105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3540 2022-12-01T10:40:42.9854256Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3541 2022-12-01T10:40:42.9854552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9854675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9854955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9855088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9855259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fngblcl 2022-12-01T10:40:42.9855801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fngblcl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9856168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9856295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9856582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9856720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9856906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp691l03pe 2022-12-01T10:40:42.9857100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp691l03pe/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9857361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9857486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9857769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9857905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9858093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxaq35l5 2022-12-01T10:40:42.9858290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxaq35l5/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9858561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9858684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9858961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9859085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9859268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8t17hzx 2022-12-01T10:40:42.9859462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8t17hzx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9859629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9859796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9859955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9860118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9860184Z ok (2.139s) 2022-12-01T10:40:42.9860189Z 2022-12-01T10:40:42.9860374Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9860453Z Ran 1 test in 2.139s 2022-12-01T10:40:42.9860458Z 2022-12-01T10:40:42.9860517Z OK 2022-12-01T10:40:42.9860521Z 2022-12-01T10:40:42.9860603Z Generating XML reports... 2022-12-01T10:40:42.9860943Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103011.xml 2022-12-01T10:40:42.9861211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9861382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9861664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9861798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9861970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9ycxaij 2022-12-01T10:40:42.9862166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9ycxaij/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9862448Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9862453Z 2022-12-01T10:40:42.9862527Z Running tests... 2022-12-01T10:40:42.9862722Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9862949Z test_py_class_instance_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3761 2022-12-01T10:40:42.9863110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3762 2022-12-01T10:40:42.9863267Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3763 2022-12-01T10:40:42.9863406Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3764 2022-12-01T10:40:42.9863680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9863808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9864093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9864230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9864416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ijk_x3k 2022-12-01T10:40:42.9864612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ijk_x3k/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9864881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9865007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9865278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9865413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9865596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h_379it 2022-12-01T10:40:42.9865789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h_379it/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9866059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9866182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9866461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9866595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9866779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby2wtddw 2022-12-01T10:40:42.9866962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby2wtddw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9867232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9867354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9867629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9867764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9867978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9o_kvdn 2022-12-01T10:40:42.9868171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9o_kvdn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9868333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9868484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9868644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9868832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9868901Z ok (2.139s) 2022-12-01T10:40:42.9868907Z 2022-12-01T10:40:42.9869104Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9869178Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9869184Z 2022-12-01T10:40:42.9869242Z OK 2022-12-01T10:40:42.9869247Z 2022-12-01T10:40:42.9869333Z Generating XML reports... 2022-12-01T10:40:42.9869674Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103015.xml 2022-12-01T10:40:42.9869937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9870058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9870334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9870472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9870657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn7zeav_ 2022-12-01T10:40:42.9870852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn7zeav_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9871107Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9871112Z 2022-12-01T10:40:42.9871187Z Running tests... 2022-12-01T10:40:42.9871369Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9871592Z test_py_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3984 2022-12-01T10:40:42.9871746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3985 2022-12-01T10:40:42.9871902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3986 2022-12-01T10:40:42.9872054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3987 2022-12-01T10:40:42.9872329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9872451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9872734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9872868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9873039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfi6axbz 2022-12-01T10:40:42.9873236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfi6axbz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9873506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9873631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9873912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9874044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9874227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ezw6b9c 2022-12-01T10:40:42.9874450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ezw6b9c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9874708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9874828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9875104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9875265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9875447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_4ylbm3 2022-12-01T10:40:42.9875638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_4ylbm3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9875906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9876028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9876309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9876428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9876609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxvwfpfp 2022-12-01T10:40:42.9876811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxvwfpfp/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9876976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9877139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9877299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9877462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9877530Z ok (2.039s) 2022-12-01T10:40:42.9877535Z 2022-12-01T10:40:42.9877721Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9877797Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9877802Z 2022-12-01T10:40:42.9877862Z OK 2022-12-01T10:40:42.9877866Z 2022-12-01T10:40:42.9877952Z Generating XML reports... 2022-12-01T10:40:42.9878431Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103018.xml 2022-12-01T10:40:42.9878707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9878832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9879115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9879254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9879426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsn9f1wep 2022-12-01T10:40:42.9879621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsn9f1wep/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9879874Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9879879Z 2022-12-01T10:40:42.9879956Z Running tests... 2022-12-01T10:40:42.9880158Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9880383Z test_py_class_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4207 2022-12-01T10:40:42.9880537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4208 2022-12-01T10:40:42.9880690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4209 2022-12-01T10:40:42.9880882Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4210 2022-12-01T10:40:42.9881165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9881290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9881568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9881703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9881923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vrrfbx0 2022-12-01T10:40:42.9882117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vrrfbx0/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9882389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9882512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9882781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9882916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9883100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwk1ukd7 2022-12-01T10:40:42.9883294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwk1ukd7/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9883565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9883688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9883968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9884101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9884287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rafxicw 2022-12-01T10:40:42.9884467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rafxicw/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9884739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9884861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9885143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9885274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9885457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0gan01s 2022-12-01T10:40:42.9885646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0gan01s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9885812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9885964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9886125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9886285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9886354Z ok (2.140s) 2022-12-01T10:40:42.9886360Z 2022-12-01T10:40:42.9886556Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9886633Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9886638Z 2022-12-01T10:40:42.9886697Z OK 2022-12-01T10:40:42.9886702Z 2022-12-01T10:40:42.9886786Z Generating XML reports... 2022-12-01T10:40:42.9887112Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103022.xml 2022-12-01T10:40:42.9887423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9887546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9887884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9888021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9888208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqhmpkeo 2022-12-01T10:40:42.9888520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqhmpkeo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9888780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9888786Z 2022-12-01T10:40:42.9888858Z Running tests... 2022-12-01T10:40:42.9889042Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9889269Z test_py_function_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4430 2022-12-01T10:40:42.9889427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4431 2022-12-01T10:40:42.9889581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4432 2022-12-01T10:40:42.9889734Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4433 2022-12-01T10:40:42.9890008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9890133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9890414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9890552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9890730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcdpzthn 2022-12-01T10:40:42.9890930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcdpzthn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9891201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9891326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9891605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9891743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9891929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpka6iws7x 2022-12-01T10:40:42.9892126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpka6iws7x/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9892384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9892509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9892787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9892920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9893103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9avlpcsu 2022-12-01T10:40:42.9893300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9avlpcsu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9893568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9893692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9893972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9894127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9894310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebu3sbrl 2022-12-01T10:40:42.9894506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebu3sbrl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9894669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9894832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9895023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9895182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9895277Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.9895456Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-12-01T10:40:42.9895551Z Traceback (most recent call last): 2022-12-01T10:40:42.9895812Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9895948Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9896079Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-12-01T10:40:42.9896084Z 2022-12-01T10:40:42.9896172Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.9896361Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-12-01T10:40:42.9896453Z Traceback (most recent call last): 2022-12-01T10:40:42.9896702Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9896836Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9896962Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-12-01T10:40:42.9896967Z 2022-12-01T10:40:42.9897056Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.9897247Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-12-01T10:40:42.9897335Z Traceback (most recent call last): 2022-12-01T10:40:42.9897595Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9897724Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9897839Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-12-01T10:40:42.9897857Z 2022-12-01T10:40:42.9897932Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.9898120Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-12-01T10:40:42.9898209Z Traceback (most recent call last): 2022-12-01T10:40:42.9898465Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9898598Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9898727Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-12-01T10:40:42.9898732Z 2022-12-01T10:40:42.9898800Z ok (2.140s) 2022-12-01T10:40:42.9898805Z 2022-12-01T10:40:42.9899003Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9899066Z Ran 1 test in 2.141s 2022-12-01T10:40:42.9899071Z 2022-12-01T10:40:42.9899131Z OK 2022-12-01T10:40:42.9899136Z 2022-12-01T10:40:42.9899222Z Generating XML reports... 2022-12-01T10:40:42.9899568Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103025.xml 2022-12-01T10:40:42.9899841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9899966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9900248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9900413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9900586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7m_a11s 2022-12-01T10:40:42.9900781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7m_a11s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9901034Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9901065Z 2022-12-01T10:40:42.9901136Z Running tests... 2022-12-01T10:40:42.9901334Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9901554Z test_py_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4653 2022-12-01T10:40:42.9901708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4654 2022-12-01T10:40:42.9901859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4655 2022-12-01T10:40:42.9902012Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4656 2022-12-01T10:40:42.9902272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9902395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9902674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9902811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9902999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckhw7jic 2022-12-01T10:40:42.9903267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9903389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9903670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9903791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9903974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckaoa1dy 2022-12-01T10:40:42.9904171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckhw7jic/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9904361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckaoa1dy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9904632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9904754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9905032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9905168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9905351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhr32wda 2022-12-01T10:40:42.9905533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhr32wda/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9905805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9905925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9906206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9906338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9906519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8abb4mt8 2022-12-01T10:40:42.9906711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8abb4mt8/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9906918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9907069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9907228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9907388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9907486Z ok (2.037s) 2022-12-01T10:40:42.9907492Z 2022-12-01T10:40:42.9907692Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9907767Z Ran 1 test in 2.038s 2022-12-01T10:40:42.9907772Z 2022-12-01T10:40:42.9907834Z OK 2022-12-01T10:40:42.9907838Z 2022-12-01T10:40:42.9907922Z Generating XML reports... 2022-12-01T10:40:42.9908261Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103029.xml 2022-12-01T10:40:42.9908524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9908648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9908930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9909066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9909253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdd3nc6hj 2022-12-01T10:40:42.9909452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdd3nc6hj/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9909708Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9909713Z 2022-12-01T10:40:42.9909786Z Running tests... 2022-12-01T10:40:42.9909969Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9910197Z test_py_nested_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4876 2022-12-01T10:40:42.9910351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4877 2022-12-01T10:40:42.9910504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4878 2022-12-01T10:40:42.9910653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4879 2022-12-01T10:40:42.9910934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9911057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9911334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9911468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9911641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rmum6w1 2022-12-01T10:40:42.9911843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rmum6w1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9912113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9912238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9912517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9912653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9912836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxloc_rnl 2022-12-01T10:40:42.9913033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxloc_rnl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9913337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9913449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9913727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9913860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9914044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpta6b04vb 2022-12-01T10:40:42.9914265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpta6b04vb/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9914537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9914661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9914944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9915069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9915253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3aqkh3s 2022-12-01T10:40:42.9915449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3aqkh3s/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9915613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9915780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9915940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9916101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9916169Z ok (2.238s) 2022-12-01T10:40:42.9916174Z 2022-12-01T10:40:42.9916373Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9916437Z Ran 1 test in 2.238s 2022-12-01T10:40:42.9916442Z 2022-12-01T10:40:42.9916501Z OK 2022-12-01T10:40:42.9916506Z 2022-12-01T10:40:42.9916588Z Generating XML reports... 2022-12-01T10:40:42.9916928Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103032.xml 2022-12-01T10:40:42.9917200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9917328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9917605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9917740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9917913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk56uh3aq 2022-12-01T10:40:42.9918111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk56uh3aq/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9918485Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9918491Z 2022-12-01T10:40:42.9918563Z Running tests... 2022-12-01T10:40:42.9918760Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9918982Z test_py_no_return_result (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5099 2022-12-01T10:40:42.9919142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5100 2022-12-01T10:40:42.9919297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5101 2022-12-01T10:40:42.9919450Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5102 2022-12-01T10:40:42.9919711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9919880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9920165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9920303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9920490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1t98y00j 2022-12-01T10:40:42.9920685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1t98y00j/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9920997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9921121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9921385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9921525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9921710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_ahzvhu 2022-12-01T10:40:42.9921903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_ahzvhu/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9922175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9922298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9922578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9922711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9922892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaed5tqui 2022-12-01T10:40:42.9923073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaed5tqui/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9923345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9923465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9923743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9923880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9924064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuh4rz7ml 2022-12-01T10:40:42.9924256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuh4rz7ml/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9924417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9924567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9924732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9924892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9924961Z do nothing 2022-12-01T10:40:42.9925028Z do nothing 2022-12-01T10:40:42.9925094Z do nothing 2022-12-01T10:40:42.9925157Z do nothing 2022-12-01T10:40:42.9925207Z do nothing 2022-12-01T10:40:42.9925271Z do nothing 2022-12-01T10:40:42.9925334Z do nothing 2022-12-01T10:40:42.9925398Z do nothing 2022-12-01T10:40:42.9925464Z ok (2.038s) 2022-12-01T10:40:42.9925469Z 2022-12-01T10:40:42.9925665Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9925738Z Ran 1 test in 2.039s 2022-12-01T10:40:42.9925743Z 2022-12-01T10:40:42.9925804Z OK 2022-12-01T10:40:42.9925808Z 2022-12-01T10:40:42.9925879Z Generating XML reports... 2022-12-01T10:40:42.9926220Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103036.xml 2022-12-01T10:40:42.9926522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9926647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9926926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9927061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9927276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz67mvsui 2022-12-01T10:40:42.9927468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz67mvsui/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9927766Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9927787Z 2022-12-01T10:40:42.9927851Z Running tests... 2022-12-01T10:40:42.9928055Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9928279Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5322 2022-12-01T10:40:42.9928437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5323 2022-12-01T10:40:42.9928591Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5324 2022-12-01T10:40:42.9928745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5325 2022-12-01T10:40:42.9929020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9929146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9929414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9929547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9929736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xprqdzo 2022-12-01T10:40:42.9929932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xprqdzo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9930202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9930323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9930605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9930739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9930910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78wqgrqn 2022-12-01T10:40:42.9931101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78wqgrqn/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9931371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9931493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9931769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9931902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9932089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppuxpj50z 2022-12-01T10:40:42.9932283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppuxpj50z/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9932555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9932665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9932981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9933118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9933300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbysqme4 2022-12-01T10:40:42.9933494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbysqme4/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9933655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9933849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9934012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9934160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9934336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:42.9934516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:42.9934694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:42.9934993Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9935163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:42.9935470Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9935768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9936059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:42.9936126Z ok (2.138s) 2022-12-01T10:40:42.9936134Z 2022-12-01T10:40:42.9936319Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9936395Z Ran 1 test in 2.138s 2022-12-01T10:40:42.9936400Z 2022-12-01T10:40:42.9936459Z OK 2022-12-01T10:40:42.9936464Z 2022-12-01T10:40:42.9936548Z Generating XML reports... 2022-12-01T10:40:42.9936892Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103040.xml 2022-12-01T10:40:42.9937168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9937290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9937571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9937693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9937881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdly8qwue 2022-12-01T10:40:42.9938079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdly8qwue/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9938331Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9938337Z 2022-12-01T10:40:42.9938408Z Running tests... 2022-12-01T10:40:42.9938605Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9938842Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5557 2022-12-01T10:40:42.9938996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5558 2022-12-01T10:40:42.9939149Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5559 2022-12-01T10:40:42.9939287Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5560 2022-12-01T10:40:42.9939593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9939715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9939999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9940135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9940345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbkb4ssl 2022-12-01T10:40:42.9940540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbkb4ssl/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9940811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9940920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9941199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9941333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9941515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1onlcwgi 2022-12-01T10:40:42.9941712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1onlcwgi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9941984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9942106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9942384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9942525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9942778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo055038g 2022-12-01T10:40:42.9942979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo055038g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9943253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9943377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9943657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9943794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9943979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy3mhe9i 2022-12-01T10:40:42.9944171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy3mhe9i/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9944323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9944489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9944651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9944809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9944900Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:42.9945111Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-12-01T10:40:42.9945204Z Traceback (most recent call last): 2022-12-01T10:40:42.9945470Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9945602Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9945884Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-12-01T10:40:42.9946028Z raise ValueError(expected_err_escape) 2022-12-01T10:40:42.9946105Z ValueError: 2022-12-01T10:40:42.9946182Z First line of error 2022-12-01T10:40:42.9946255Z next line of error 2022-12-01T10:40:42.9946330Z last line of error 2022-12-01T10:40:42.9946335Z 2022-12-01T10:40:42.9946425Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:42.9946626Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-12-01T10:40:42.9946718Z Traceback (most recent call last): 2022-12-01T10:40:42.9947004Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9947137Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9947433Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-12-01T10:40:42.9947541Z raise ValueError(expected_err_escape) 2022-12-01T10:40:42.9947612Z ValueError: 2022-12-01T10:40:42.9947689Z First line of error 2022-12-01T10:40:42.9947749Z next line of error 2022-12-01T10:40:42.9947824Z last line of error 2022-12-01T10:40:42.9947829Z 2022-12-01T10:40:42.9947920Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:42.9948134Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-12-01T10:40:42.9948223Z Traceback (most recent call last): 2022-12-01T10:40:42.9948483Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9948617Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9948900Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-12-01T10:40:42.9949000Z raise ValueError(expected_err_escape) 2022-12-01T10:40:42.9949069Z ValueError: 2022-12-01T10:40:42.9949141Z First line of error 2022-12-01T10:40:42.9949212Z next line of error 2022-12-01T10:40:42.9949284Z last line of error 2022-12-01T10:40:42.9949289Z 2022-12-01T10:40:42.9949377Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:42.9949569Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-12-01T10:40:42.9949660Z Traceback (most recent call last): 2022-12-01T10:40:42.9949918Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:42.9950050Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:42.9950345Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-12-01T10:40:42.9950444Z raise ValueError(expected_err_escape) 2022-12-01T10:40:42.9950515Z ValueError: 2022-12-01T10:40:42.9950590Z First line of error 2022-12-01T10:40:42.9950651Z next line of error 2022-12-01T10:40:42.9950723Z last line of error 2022-12-01T10:40:42.9950729Z 2022-12-01T10:40:42.9950795Z ok (2.140s) 2022-12-01T10:40:42.9950802Z 2022-12-01T10:40:42.9951001Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9951074Z Ran 1 test in 2.140s 2022-12-01T10:40:42.9951078Z 2022-12-01T10:40:42.9951142Z OK 2022-12-01T10:40:42.9951146Z 2022-12-01T10:40:42.9951235Z Generating XML reports... 2022-12-01T10:40:42.9951576Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103043.xml 2022-12-01T10:40:42.9951838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9951963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9952250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9952384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9952638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiz7zbpz1 2022-12-01T10:40:42.9952838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiz7zbpz1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9953096Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9953101Z 2022-12-01T10:40:42.9953173Z Running tests... 2022-12-01T10:40:42.9953375Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9953616Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5780 2022-12-01T10:40:42.9953773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5781 2022-12-01T10:40:42.9953930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5782 2022-12-01T10:40:42.9954083Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5783 2022-12-01T10:40:42.9954367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9954493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9954778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9954915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9955091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ijdnwe_ 2022-12-01T10:40:42.9955292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ijdnwe_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9955568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9955698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9955983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9956119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9956308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb23ue5_ 2022-12-01T10:40:42.9956505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb23ue5_/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9956778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9956892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9957175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9957310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9957495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjr5i8ul 2022-12-01T10:40:42.9957688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjr5i8ul/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9957960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9958085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9958494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9958619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9958807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp641o0ohg 2022-12-01T10:40:42.9959004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp641o0ohg/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9959169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9959429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9959595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9959759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9959829Z ok (2.138s) 2022-12-01T10:40:42.9959836Z 2022-12-01T10:40:42.9960039Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9960144Z Ran 1 test in 2.138s 2022-12-01T10:40:42.9960149Z 2022-12-01T10:40:42.9960212Z OK 2022-12-01T10:40:42.9960217Z 2022-12-01T10:40:42.9960304Z Generating XML reports... 2022-12-01T10:40:42.9960651Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103047.xml 2022-12-01T10:40:42.9960932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9961063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9961350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9961516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9961692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph36qtii1 2022-12-01T10:40:42.9961893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph36qtii1/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9962150Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9962155Z 2022-12-01T10:40:42.9962227Z Running tests... 2022-12-01T10:40:42.9962425Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9962642Z test_py_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6003 2022-12-01T10:40:42.9962796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6004 2022-12-01T10:40:42.9962950Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6005 2022-12-01T10:40:42.9963101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6006 2022-12-01T10:40:42.9963363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9963488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9963770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9963904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9964091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5hf8bvh 2022-12-01T10:40:42.9964292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5hf8bvh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9964568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9964693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9964973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9965098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9965280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0sflz5hi 2022-12-01T10:40:42.9965476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0sflz5hi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9965749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9965928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9966537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9966677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9966865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0hm5ryz 2022-12-01T10:40:42.9967093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0hm5ryz/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9967437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9967561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9967901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9968038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9968232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1bdvvdh 2022-12-01T10:40:42.9968429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1bdvvdh/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9968600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9968780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9968941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9969106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9969179Z ok (2.142s) 2022-12-01T10:40:42.9969187Z 2022-12-01T10:40:42.9969388Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9969465Z Ran 1 test in 2.142s 2022-12-01T10:40:42.9969470Z 2022-12-01T10:40:42.9969530Z OK 2022-12-01T10:40:42.9969534Z 2022-12-01T10:40:42.9969623Z Generating XML reports... 2022-12-01T10:40:42.9969969Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103050.xml 2022-12-01T10:40:42.9970231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9970355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9970642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9970775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9970961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsa26o29n 2022-12-01T10:40:42.9971161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsa26o29n/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9971419Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9971425Z 2022-12-01T10:40:42.9971502Z Running tests... 2022-12-01T10:40:42.9971699Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9971914Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6226 2022-12-01T10:40:42.9972070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6227 2022-12-01T10:40:42.9972228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6228 2022-12-01T10:40:42.9972385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6229 2022-12-01T10:40:42.9972660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9972786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9973118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9973258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9973434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0kzmsom6 2022-12-01T10:40:42.9973630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0kzmsom6/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9973932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9974055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9974345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9974480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9974663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqs8ehg9 2022-12-01T10:40:42.9974861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqs8ehg9/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9975138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9975250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9975532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9975671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9975856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtlekpnf 2022-12-01T10:40:42.9976053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtlekpnf/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9976333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9976459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9976738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9976872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9977042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm6r713c 2022-12-01T10:40:42.9977240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm6r713c/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9977410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9977573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9977735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9977902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9977974Z ok (2.244s) 2022-12-01T10:40:42.9977979Z 2022-12-01T10:40:42.9978178Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9978245Z Ran 1 test in 2.245s 2022-12-01T10:40:42.9978249Z 2022-12-01T10:40:42.9978310Z OK 2022-12-01T10:40:42.9978315Z 2022-12-01T10:40:42.9978400Z Generating XML reports... 2022-12-01T10:40:42.9978741Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103054.xml 2022-12-01T10:40:42.9979021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9979144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9979428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9979605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9979792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskjdpzbc 2022-12-01T10:40:42.9979978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskjdpzbc/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9980238Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9980243Z 2022-12-01T10:40:42.9980352Z Running tests... 2022-12-01T10:40:42.9980555Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9980773Z test_py_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6449 2022-12-01T10:40:42.9980932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6450 2022-12-01T10:40:42.9981085Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6451 2022-12-01T10:40:42.9981244Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6452 2022-12-01T10:40:42.9981506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9981630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9981913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9982054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9982240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zzcbyxo 2022-12-01T10:40:42.9982440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zzcbyxo/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9982730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9982862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9983145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9983315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9983572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo052pjjy 2022-12-01T10:40:42.9983769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo052pjjy/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9984055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9984178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9984461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9984600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9984794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgz8308_i 2022-12-01T10:40:42.9984982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgz8308_i/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9985263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9985395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9985684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9985820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9986012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqitsyrfi 2022-12-01T10:40:42.9986210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqitsyrfi/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9986419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9986585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9986736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9986896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9986963Z ok (2.141s) 2022-12-01T10:40:42.9986998Z 2022-12-01T10:40:42.9987197Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9987272Z Ran 1 test in 2.141s 2022-12-01T10:40:42.9987277Z 2022-12-01T10:40:42.9987340Z OK 2022-12-01T10:40:42.9987345Z 2022-12-01T10:40:42.9987428Z Generating XML reports... 2022-12-01T10:40:42.9987777Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103058.xml 2022-12-01T10:40:42.9988046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9988169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9988452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9988587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9988775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjihyifa3 2022-12-01T10:40:42.9988978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjihyifa3/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9989239Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9989244Z 2022-12-01T10:40:42.9989320Z Running tests... 2022-12-01T10:40:42.9989519Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9989738Z test_py_tensors_in_container (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6672 2022-12-01T10:40:42.9989896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6673 2022-12-01T10:40:42.9990047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6674 2022-12-01T10:40:42.9990205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6675 2022-12-01T10:40:42.9990486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9990612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9990893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9991028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9991203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolqv0w6u 2022-12-01T10:40:42.9991403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolqv0w6u/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9991674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9991797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9992077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9992217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9992401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmu0xzm1g 2022-12-01T10:40:42.9992597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmu0xzm1g/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9992899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9993013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9993298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9993435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9993619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bg6dzpx 2022-12-01T10:40:42.9993847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bg6dzpx/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9994212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9994383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9994722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9994863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9995034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7o4s8we 2022-12-01T10:40:42.9995229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7o4s8we/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9995399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:42.9995575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:42.9995741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:42.9995907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:42.9995975Z ok (2.044s) 2022-12-01T10:40:42.9995980Z 2022-12-01T10:40:42.9996182Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9996247Z Ran 1 test in 2.044s 2022-12-01T10:40:42.9996253Z 2022-12-01T10:40:42.9996311Z OK 2022-12-01T10:40:42.9996317Z 2022-12-01T10:40:42.9996403Z Generating XML reports... 2022-12-01T10:40:42.9996745Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103101.xml 2022-12-01T10:40:42.9997019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9997143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:42.9997430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:42.9997565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:42.9997736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx8z4gs2 2022-12-01T10:40:42.9997935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx8z4gs2/_remote_module_non_scriptable.py 2022-12-01T10:40:42.9998333Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:42.9998340Z 2022-12-01T10:40:42.9998416Z Running tests... 2022-12-01T10:40:42.9998620Z ---------------------------------------------------------------------- 2022-12-01T10:40:42.9998852Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6895 2022-12-01T10:40:42.9999015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6896 2022-12-01T10:40:42.9999169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6897 2022-12-01T10:40:42.9999321Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6898 2022-12-01T10:40:42.9999587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:42.9999815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0000104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0000242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0000429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcq4wo6b 2022-12-01T10:40:43.0000624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcq4wo6b/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0000943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0001065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0001348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0001473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0001657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uv559w4 2022-12-01T10:40:43.0001852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uv559w4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0002130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0002251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0002531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0002671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0002854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimhoelea 2022-12-01T10:40:43.0003037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimhoelea/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0003311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0003431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0003712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0003845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0004031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqj1al6ob 2022-12-01T10:40:43.0004222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqj1al6ob/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0004384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0004545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0004698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0004859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0004924Z ok (2.241s) 2022-12-01T10:40:43.0004929Z 2022-12-01T10:40:43.0005127Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0005201Z Ran 1 test in 2.241s 2022-12-01T10:40:43.0005206Z 2022-12-01T10:40:43.0005266Z OK 2022-12-01T10:40:43.0005273Z 2022-12-01T10:40:43.0005360Z Generating XML reports... 2022-12-01T10:40:43.0005703Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103105.xml 2022-12-01T10:40:43.0005964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0006088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0006400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0006537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0006721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2q7rjow 2022-12-01T10:40:43.0006914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2q7rjow/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0007172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0007209Z 2022-12-01T10:40:43.0007280Z Running tests... 2022-12-01T10:40:43.0007479Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0007745Z test_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7118 2022-12-01T10:40:43.0007906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7119 2022-12-01T10:40:43.0008065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7120 2022-12-01T10:40:43.0008216Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7121 2022-12-01T10:40:43.0008495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0008619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0008904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0009043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0009214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpny_vwync 2022-12-01T10:40:43.0009413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpny_vwync/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0009689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0009814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0010095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0010232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0010416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_xz_1nb 2022-12-01T10:40:43.0010609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_xz_1nb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0010878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0010989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0011271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0011405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0011587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7bcqndf 2022-12-01T10:40:43.0011783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7bcqndf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0012056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0012182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0012461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0012595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0012769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijeiak5t 2022-12-01T10:40:43.0013006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijeiak5t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0013170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0013333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0013491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0013654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0013746Z ok (2.039s) 2022-12-01T10:40:43.0013751Z 2022-12-01T10:40:43.0013948Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0014013Z Ran 1 test in 2.039s 2022-12-01T10:40:43.0014018Z 2022-12-01T10:40:43.0014076Z OK 2022-12-01T10:40:43.0014081Z 2022-12-01T10:40:43.0014163Z Generating XML reports... 2022-12-01T10:40:43.0014503Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103108.xml 2022-12-01T10:40:43.0014777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0014903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0015181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0015318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0015489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimra681d 2022-12-01T10:40:43.0015682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimra681d/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0015933Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0015938Z 2022-12-01T10:40:43.0016011Z Running tests... 2022-12-01T10:40:43.0016212Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0016434Z test_py_user_defined (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7341 2022-12-01T10:40:43.0016588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7342 2022-12-01T10:40:43.0016739Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7343 2022-12-01T10:40:43.0016892Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7344 2022-12-01T10:40:43.0017152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0017274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0017558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0017694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0017878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxu13003 2022-12-01T10:40:43.0018071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxu13003/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0018345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0018470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0018755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0018882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0019067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvppko0vz 2022-12-01T10:40:43.0019265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvppko0vz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0019569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0019695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0019978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0020120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0020334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxik0vius 2022-12-01T10:40:43.0020519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxik0vius/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0020793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0020915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0021195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0021330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0021516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdisfvgdj 2022-12-01T10:40:43.0021710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdisfvgdj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0021880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0022044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0022196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0022355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0022421Z ok (2.142s) 2022-12-01T10:40:43.0022427Z 2022-12-01T10:40:43.0022629Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0022706Z Ran 1 test in 2.142s 2022-12-01T10:40:43.0022711Z 2022-12-01T10:40:43.0022772Z OK 2022-12-01T10:40:43.0022776Z 2022-12-01T10:40:43.0022861Z Generating XML reports... 2022-12-01T10:40:43.0023202Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103112.xml 2022-12-01T10:40:43.0023463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0023589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0023868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0024002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0024189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfp3tfkd 2022-12-01T10:40:43.0024386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfp3tfkd/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0024641Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0024646Z 2022-12-01T10:40:43.0024719Z Running tests... 2022-12-01T10:40:43.0024914Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0025158Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7564 2022-12-01T10:40:43.0025313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7565 2022-12-01T10:40:43.0025470Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7566 2022-12-01T10:40:43.0025621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7567 2022-12-01T10:40:43.0026034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0026161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0026443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0026578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0026751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqja2nq8w 2022-12-01T10:40:43.0027059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0027182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0027464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0027605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0027792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0elvj1ty 2022-12-01T10:40:43.0027990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqja2nq8w/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0028184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0elvj1ty/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0028459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0028576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0028856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0028986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0029171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkavoi3y 2022-12-01T10:40:43.0029370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkavoi3y/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0029640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0029763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0030040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0030176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0030345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cqodude 2022-12-01T10:40:43.0030538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cqodude/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0030701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0030864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0031025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0031190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0031256Z ok (1.837s) 2022-12-01T10:40:43.0031261Z 2022-12-01T10:40:43.0031460Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0031525Z Ran 1 test in 1.837s 2022-12-01T10:40:43.0031530Z 2022-12-01T10:40:43.0031591Z OK 2022-12-01T10:40:43.0031596Z 2022-12-01T10:40:43.0031677Z Generating XML reports... 2022-12-01T10:40:43.0032018Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103115.xml 2022-12-01T10:40:43.0032292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0032452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0032734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0032870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0033056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1rpm67r 2022-12-01T10:40:43.0033239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1rpm67r/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0033531Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0033536Z 2022-12-01T10:40:43.0033606Z Running tests... 2022-12-01T10:40:43.0033804Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0034015Z test_reinit (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7623 2022-12-01T10:40:43.0034173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7624 2022-12-01T10:40:43.0034324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7625 2022-12-01T10:40:43.0034475Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7626 2022-12-01T10:40:43.0034738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0034862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0035146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0035281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0035465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpthh15l 2022-12-01T10:40:43.0035662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpthh15l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0035931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0036052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0036333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0036455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0036639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqflb1dea 2022-12-01T10:40:43.0036914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0037035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0037314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0037449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0037633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_gr0fon 2022-12-01T10:40:43.0037827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqflb1dea/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0038002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_gr0fon/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0038416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0038543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0038825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0038959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0039213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9zz27q0 2022-12-01T10:40:43.0039409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9zz27q0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0039572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0039733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0039883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0040079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0040259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0040440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0040617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0040792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0041097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0041400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0041696Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0041977Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0042043Z ok (2.039s) 2022-12-01T10:40:43.0042048Z 2022-12-01T10:40:43.0042245Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0042320Z Ran 1 test in 2.040s 2022-12-01T10:40:43.0042325Z 2022-12-01T10:40:43.0042389Z OK 2022-12-01T10:40:43.0042394Z 2022-12-01T10:40:43.0042476Z Generating XML reports... 2022-12-01T10:40:43.0042817Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103119.xml 2022-12-01T10:40:43.0043093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0043205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0043491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0043627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0043813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfciumkoq 2022-12-01T10:40:43.0044008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfciumkoq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0044265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0044270Z 2022-12-01T10:40:43.0044342Z Running tests... 2022-12-01T10:40:43.0044536Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0044759Z test_remote_same_worker (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7858 2022-12-01T10:40:43.0044903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7859 2022-12-01T10:40:43.0045055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7860 2022-12-01T10:40:43.0045206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7861 2022-12-01T10:40:43.0045477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0045646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0045928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0046062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0046249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbegl83hl 2022-12-01T10:40:43.0046445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbegl83hl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0046733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0046857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0047139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0047272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0047458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgg34v8z 2022-12-01T10:40:43.0047652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgg34v8z/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0047986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0048111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0048380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0048516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0048697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pi1eohs 2022-12-01T10:40:43.0048891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pi1eohs/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0049167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0049290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0049571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0049706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0049893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpviqxv6j_ 2022-12-01T10:40:43.0050076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpviqxv6j_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0050239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0050403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0050566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0050728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0050797Z ok (2.238s) 2022-12-01T10:40:43.0050802Z 2022-12-01T10:40:43.0051000Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0051074Z Ran 1 test in 2.238s 2022-12-01T10:40:43.0051080Z 2022-12-01T10:40:43.0051128Z OK 2022-12-01T10:40:43.0051132Z 2022-12-01T10:40:43.0051222Z Generating XML reports... 2022-12-01T10:40:43.0051558Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103122.xml 2022-12-01T10:40:43.0051831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0051957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0052274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0052413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0052597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0aqtixe 2022-12-01T10:40:43.0052791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0aqtixe/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0053033Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0053089Z 2022-12-01T10:40:43.0053166Z Running tests... 2022-12-01T10:40:43.0053363Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0053580Z test_remote_throw (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8081 2022-12-01T10:40:43.0053737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8082 2022-12-01T10:40:43.0053891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8083 2022-12-01T10:40:43.0054045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8084 2022-12-01T10:40:43.0054316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0054425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0054703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0054840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0055027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4tp3rmu 2022-12-01T10:40:43.0055224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4tp3rmu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0055495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0055621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0055902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0056037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0056206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0_6omsk 2022-12-01T10:40:43.0056400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0_6omsk/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0056672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0056797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0057075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0057212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0057400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogrmsi5b 2022-12-01T10:40:43.0057592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogrmsi5b/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0057851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0057976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0058258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0058391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0058572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptse_1cow 2022-12-01T10:40:43.0058792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptse_1cow/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0058958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0059121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0059282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0059429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0059552Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0059670Z ValueError('Expected error') 2022-12-01T10:40:43.0059762Z Traceback (most recent call last): 2022-12-01T10:40:43.0060021Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0060157Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0060449Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-12-01T10:40:43.0060546Z raise ValueError("Expected error") 2022-12-01T10:40:43.0060620Z ValueError: Expected error 2022-12-01T10:40:43.0060625Z 2022-12-01T10:40:43.0060717Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0060836Z ValueError('Expected error') 2022-12-01T10:40:43.0060926Z Traceback (most recent call last): 2022-12-01T10:40:43.0061185Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0061319Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0061603Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-12-01T10:40:43.0061696Z raise ValueError("Expected error") 2022-12-01T10:40:43.0061768Z ValueError: Expected error 2022-12-01T10:40:43.0061773Z 2022-12-01T10:40:43.0061863Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0061982Z ValueError('Expected error') 2022-12-01T10:40:43.0062075Z Traceback (most recent call last): 2022-12-01T10:40:43.0062337Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0062470Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0062752Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-12-01T10:40:43.0062837Z raise ValueError("Expected error") 2022-12-01T10:40:43.0062919Z ValueError: Expected error 2022-12-01T10:40:43.0062924Z 2022-12-01T10:40:43.0063012Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0063129Z ValueError('Expected error') 2022-12-01T10:40:43.0063219Z Traceback (most recent call last): 2022-12-01T10:40:43.0063478Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0063611Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0063896Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-12-01T10:40:43.0063976Z raise ValueError("Expected error") 2022-12-01T10:40:43.0064060Z ValueError: Expected error 2022-12-01T10:40:43.0064065Z 2022-12-01T10:40:43.0064129Z ok (2.144s) 2022-12-01T10:40:43.0064134Z 2022-12-01T10:40:43.0064332Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0064409Z Ran 1 test in 2.144s 2022-12-01T10:40:43.0064415Z 2022-12-01T10:40:43.0064476Z OK 2022-12-01T10:40:43.0064481Z 2022-12-01T10:40:43.0064565Z Generating XML reports... 2022-12-01T10:40:43.0064904Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103126.xml 2022-12-01T10:40:43.0065207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0065320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0065603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0065739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0065925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwwu8hw5 2022-12-01T10:40:43.0066152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwwu8hw5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0066407Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0066413Z 2022-12-01T10:40:43.0066484Z Running tests... 2022-12-01T10:40:43.0066681Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0066897Z test_remote_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8304 2022-12-01T10:40:43.0067054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8305 2022-12-01T10:40:43.0067205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8306 2022-12-01T10:40:43.0067355Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8307 2022-12-01T10:40:43.0067630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0067755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0068037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0068171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0068355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7b1vl82c 2022-12-01T10:40:43.0068539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7b1vl82c/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0068808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0068931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0069210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0069345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0069528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5e86dq_s 2022-12-01T10:40:43.0069719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5e86dq_s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0069990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0070100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0070376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0070511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0070692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprycc584l 2022-12-01T10:40:43.0070884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprycc584l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0071156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0071278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0071558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0071724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0071894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal9g4do7 2022-12-01T10:40:43.0072090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal9g4do7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0072255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0072418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0072606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0072767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0072858Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0073049Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-12-01T10:40:43.0073128Z Traceback (most recent call last): 2022-12-01T10:40:43.0073389Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0073522Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0073652Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-12-01T10:40:43.0073657Z 2022-12-01T10:40:43.0073746Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0073865Z ValueError('Expected error') 2022-12-01T10:40:43.0073954Z Traceback (most recent call last): 2022-12-01T10:40:43.0074212Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0074329Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0074609Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0074699Z raise ValueError(expected_err) 2022-12-01T10:40:43.0074789Z ValueError: Expected error 2022-12-01T10:40:43.0074797Z 2022-12-01T10:40:43.0074886Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0075005Z ValueError('Expected error') 2022-12-01T10:40:43.0075095Z Traceback (most recent call last): 2022-12-01T10:40:43.0075354Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0075472Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0075751Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0075846Z raise ValueError(expected_err) 2022-12-01T10:40:43.0075930Z ValueError: Expected error 2022-12-01T10:40:43.0075935Z 2022-12-01T10:40:43.0076024Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0076213Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-12-01T10:40:43.0076301Z Traceback (most recent call last): 2022-12-01T10:40:43.0076558Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0076676Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0076803Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-12-01T10:40:43.0076807Z 2022-12-01T10:40:43.0076896Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0077014Z ValueError('Expected error') 2022-12-01T10:40:43.0077102Z Traceback (most recent call last): 2022-12-01T10:40:43.0077358Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0077487Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0077767Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0077845Z raise ValueError(expected_err) 2022-12-01T10:40:43.0077930Z ValueError: Expected error 2022-12-01T10:40:43.0077935Z 2022-12-01T10:40:43.0078053Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0078592Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-12-01T10:40:43.0078701Z Traceback (most recent call last): 2022-12-01T10:40:43.0079302Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0079444Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0079650Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-12-01T10:40:43.0079671Z 2022-12-01T10:40:43.0079749Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0079876Z ValueError('Expected error') 2022-12-01T10:40:43.0079969Z Traceback (most recent call last): 2022-12-01T10:40:43.0080227Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0080358Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0080648Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0080739Z raise ValueError(expected_err) 2022-12-01T10:40:43.0080811Z ValueError: Expected error 2022-12-01T10:40:43.0080830Z 2022-12-01T10:40:43.0080905Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0081093Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-12-01T10:40:43.0081188Z Traceback (most recent call last): 2022-12-01T10:40:43.0081444Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0081597Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0081778Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-12-01T10:40:43.0081783Z 2022-12-01T10:40:43.0081850Z ok (2.245s) 2022-12-01T10:40:43.0081854Z 2022-12-01T10:40:43.0082055Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0082118Z Ran 1 test in 2.245s 2022-12-01T10:40:43.0082122Z 2022-12-01T10:40:43.0082209Z OK 2022-12-01T10:40:43.0082216Z 2022-12-01T10:40:43.0082350Z Generating XML reports... 2022-12-01T10:40:43.0082745Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103129.xml 2022-12-01T10:40:43.0083095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0083267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0083710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0083850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0084025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0y41olzr 2022-12-01T10:40:43.0084224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0y41olzr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0084483Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0084489Z 2022-12-01T10:40:43.0084563Z Running tests... 2022-12-01T10:40:43.0084764Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0084985Z test_return_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8527 2022-12-01T10:40:43.0085144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8528 2022-12-01T10:40:43.0085297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8529 2022-12-01T10:40:43.0085449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8530 2022-12-01T10:40:43.0085775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0085902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0086183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0086318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0086503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mr77a1j 2022-12-01T10:40:43.0086730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mr77a1j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0087001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0087126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0087392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0087534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0087778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgml9op0 2022-12-01T10:40:43.0087981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgml9op0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0088257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0088382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0088667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0088805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0088990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2o2i8nva 2022-12-01T10:40:43.0089173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2o2i8nva/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0089444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0089566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0089844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0089979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0090169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmqjse34 2022-12-01T10:40:43.0090367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmqjse34/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0090534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0090686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0090852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0091016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0091082Z ok (2.140s) 2022-12-01T10:40:43.0091087Z 2022-12-01T10:40:43.0091289Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0091364Z Ran 1 test in 2.141s 2022-12-01T10:40:43.0091371Z 2022-12-01T10:40:43.0091432Z OK 2022-12-01T10:40:43.0091437Z 2022-12-01T10:40:43.0091521Z Generating XML reports... 2022-12-01T10:40:43.0091860Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103133.xml 2022-12-01T10:40:43.0092122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0092247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0092590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0092726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0092908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4t_l6bym 2022-12-01T10:40:43.0093097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4t_l6bym/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0093395Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0093401Z 2022-12-01T10:40:43.0093472Z Running tests... 2022-12-01T10:40:43.0093655Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0093875Z test_return_future_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8750 2022-12-01T10:40:43.0094033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8751 2022-12-01T10:40:43.0094188Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8752 2022-12-01T10:40:43.0094341Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8753 2022-12-01T10:40:43.0094616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0094739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0095019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0095153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0095324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsdx48u4 2022-12-01T10:40:43.0095521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsdx48u4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0095792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0095915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0096193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0096325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0096511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqq6k81f 2022-12-01T10:40:43.0096707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqq6k81f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0096976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0097086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0097363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0097497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0097680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqnxco51 2022-12-01T10:40:43.0097872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqnxco51/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0098139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0098265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0098540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0098661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0098846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8b9tzt7 2022-12-01T10:40:43.0099067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8b9tzt7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0099231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0099394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0099556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0099743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0099809Z ok (2.239s) 2022-12-01T10:40:43.0099815Z 2022-12-01T10:40:43.0100011Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0100074Z Ran 1 test in 2.239s 2022-12-01T10:40:43.0100079Z 2022-12-01T10:40:43.0100142Z OK 2022-12-01T10:40:43.0100147Z 2022-12-01T10:40:43.0100229Z Generating XML reports... 2022-12-01T10:40:43.0100570Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103137.xml 2022-12-01T10:40:43.0100841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0100964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0101240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0101377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0101549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwkd1_mz 2022-12-01T10:40:43.0101744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwkd1_mz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0101997Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0102003Z 2022-12-01T10:40:43.0102077Z Running tests... 2022-12-01T10:40:43.0102272Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0102496Z test_return_future_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8973 2022-12-01T10:40:43.0102649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8974 2022-12-01T10:40:43.0102801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8975 2022-12-01T10:40:43.0102955Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8976 2022-12-01T10:40:43.0103216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0103338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0103616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0103752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0103936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbw05xg1u 2022-12-01T10:40:43.0104133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbw05xg1u/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0104410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0104535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0104802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0104936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0105119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1px36a5n 2022-12-01T10:40:43.0105345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1px36a5n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0105621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0105743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0106021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0106185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0106369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy60xpc2n 2022-12-01T10:40:43.0106550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy60xpc2n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0106821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0106946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0107226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0107361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0107545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsqu9g4h 2022-12-01T10:40:43.0107740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsqu9g4h/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0107904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0108053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0108215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0108376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0108443Z ok (2.140s) 2022-12-01T10:40:43.0108451Z 2022-12-01T10:40:43.0108647Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0108724Z Ran 1 test in 2.141s 2022-12-01T10:40:43.0108729Z 2022-12-01T10:40:43.0108788Z OK 2022-12-01T10:40:43.0108793Z 2022-12-01T10:40:43.0108877Z Generating XML reports... 2022-12-01T10:40:43.0109220Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103140.xml 2022-12-01T10:40:43.0109481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0109603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0109879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0110012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0110198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphg5v5f8b 2022-12-01T10:40:43.0110393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphg5v5f8b/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0110646Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0110652Z 2022-12-01T10:40:43.0110722Z Running tests... 2022-12-01T10:40:43.0110905Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0111129Z test_return_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9196 2022-12-01T10:40:43.0111282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9197 2022-12-01T10:40:43.0111433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9198 2022-12-01T10:40:43.0111585Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9199 2022-12-01T10:40:43.0111889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0112012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0112291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0112426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0112626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaehcck4r 2022-12-01T10:40:43.0112822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaehcck4r/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0113091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0113211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0113494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0113626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0113808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyios6wy 2022-12-01T10:40:43.0114002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyios6wy/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0114260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0114386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0114662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0114795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0114979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0p0wv7t 2022-12-01T10:40:43.0115173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0p0wv7t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0115441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0115564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0115844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0115968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0116147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm42lsn1 2022-12-01T10:40:43.0116337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm42lsn1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0116498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0116664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0116824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0116985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0117052Z ok (2.140s) 2022-12-01T10:40:43.0117058Z 2022-12-01T10:40:43.0117253Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0117316Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0117321Z 2022-12-01T10:40:43.0117381Z OK 2022-12-01T10:40:43.0117386Z 2022-12-01T10:40:43.0117467Z Generating XML reports... 2022-12-01T10:40:43.0117804Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103144.xml 2022-12-01T10:40:43.0118073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0118370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0118661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0118798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0118968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp400lmrnz 2022-12-01T10:40:43.0119201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp400lmrnz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0119457Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0119463Z 2022-12-01T10:40:43.0119535Z Running tests... 2022-12-01T10:40:43.0119732Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0119954Z test_rpc_barrier_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9419 2022-12-01T10:40:43.0120113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9420 2022-12-01T10:40:43.0120263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9421 2022-12-01T10:40:43.0120416Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9422 2022-12-01T10:40:43.0120676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0120800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0121081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0121215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0121400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8s7209x 2022-12-01T10:40:43.0121597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8s7209x/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0121869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0121995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0122264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0122402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0122584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppol1uusf 2022-12-01T10:40:43.0122786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppol1uusf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0123061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0123186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0123464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0123596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0123779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_wfrz65 2022-12-01T10:40:43.0123960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_wfrz65/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0124232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0124352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0124628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0124759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0124972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktntw1m9 2022-12-01T10:40:43.0125169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktntw1m9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0125332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0125481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0125668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0125827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0125895Z ok (2.138s) 2022-12-01T10:40:43.0125900Z 2022-12-01T10:40:43.0126095Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0126169Z Ran 1 test in 2.139s 2022-12-01T10:40:43.0126174Z 2022-12-01T10:40:43.0126234Z OK 2022-12-01T10:40:43.0126239Z 2022-12-01T10:40:43.0126323Z Generating XML reports... 2022-12-01T10:40:43.0126660Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103147.xml 2022-12-01T10:40:43.0126920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0127041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0127321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0127454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0127638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5idevvww 2022-12-01T10:40:43.0127897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5idevvww/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0128160Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0128165Z 2022-12-01T10:40:43.0128238Z Running tests... 2022-12-01T10:40:43.0128421Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0128653Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9642 2022-12-01T10:40:43.0128809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9643 2022-12-01T10:40:43.0128966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9644 2022-12-01T10:40:43.0129117Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9645 2022-12-01T10:40:43.0129393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0129517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0129801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0129938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0130109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpledit_1l 2022-12-01T10:40:43.0130305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpledit_1l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0130577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0130700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0130983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0131116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0131336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rveed3y 2022-12-01T10:40:43.0131533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rveed3y/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0131793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0131917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0132196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0132358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0132541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcap0wqrv 2022-12-01T10:40:43.0132735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcap0wqrv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0133010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0133134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0133411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0133533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0133714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqo7is71v 2022-12-01T10:40:43.0133909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqo7is71v/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0134071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0134234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0134393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0134556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0134623Z ok (2.136s) 2022-12-01T10:40:43.0134628Z 2022-12-01T10:40:43.0134815Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0134890Z Ran 1 test in 2.137s 2022-12-01T10:40:43.0134895Z 2022-12-01T10:40:43.0134955Z OK 2022-12-01T10:40:43.0134960Z 2022-12-01T10:40:43.0135041Z Generating XML reports... 2022-12-01T10:40:43.0135383Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103151.xml 2022-12-01T10:40:43.0135656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0135779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0136059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0136195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0136369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9c0plrl6 2022-12-01T10:40:43.0136566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9c0plrl6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0136821Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0136829Z 2022-12-01T10:40:43.0136900Z Running tests... 2022-12-01T10:40:43.0137096Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0137326Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9877 2022-12-01T10:40:43.0137480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9878 2022-12-01T10:40:43.0137634Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9879 2022-12-01T10:40:43.0137826Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9880 2022-12-01T10:40:43.0138091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0138216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0138496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0138658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0138842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywnyg7s2 2022-12-01T10:40:43.0139038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywnyg7s2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0139311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0139434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0139700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0139834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0140016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffp6wrfg 2022-12-01T10:40:43.0140211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffp6wrfg/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0140482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0140603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0140877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0141013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0141195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm8i2zr5 2022-12-01T10:40:43.0141374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm8i2zr5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0141642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0141764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0142045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0142177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0142356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fjz2v2a 2022-12-01T10:40:43.0142551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fjz2v2a/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0142718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0142869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0143030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0143188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0143257Z ok (2.139s) 2022-12-01T10:40:43.0143263Z 2022-12-01T10:40:43.0143458Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0143534Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0143539Z 2022-12-01T10:40:43.0143596Z OK 2022-12-01T10:40:43.0143601Z 2022-12-01T10:40:43.0143684Z Generating XML reports... 2022-12-01T10:40:43.0144022Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103155.xml 2022-12-01T10:40:43.0144320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0144442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0144723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0144855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0145071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_t9p8yy 2022-12-01T10:40:43.0145267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_t9p8yy/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0145523Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0145528Z 2022-12-01T10:40:43.0145598Z Running tests... 2022-12-01T10:40:43.0145782Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0146005Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10100 2022-12-01T10:40:43.0146162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10101 2022-12-01T10:40:43.0146317Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10102 2022-12-01T10:40:43.0146468Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10103 2022-12-01T10:40:43.0146745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0146866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0147149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0147279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0147453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv49rke0e 2022-12-01T10:40:43.0147647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv49rke0e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0147919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0148040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0148317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0148451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0148634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1y1ex41s 2022-12-01T10:40:43.0148826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1y1ex41s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0149084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0149205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0149481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0149616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0149799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3gxznu1 2022-12-01T10:40:43.0150070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0150191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0150467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0150598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0150798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbak62oow 2022-12-01T10:40:43.0150991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3gxznu1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0151180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbak62oow/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0151347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0151536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0151698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0151859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0151929Z ok (2.140s) 2022-12-01T10:40:43.0151934Z 2022-12-01T10:40:43.0152122Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0152196Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0152202Z 2022-12-01T10:40:43.0152264Z OK 2022-12-01T10:40:43.0152268Z 2022-12-01T10:40:43.0152354Z Generating XML reports... 2022-12-01T10:40:43.0152690Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103158.xml 2022-12-01T10:40:43.0152961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0153086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0153363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0153496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0153667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbk5k8dh 2022-12-01T10:40:43.0153863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbk5k8dh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0154119Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0154124Z 2022-12-01T10:40:43.0154197Z Running tests... 2022-12-01T10:40:43.0154396Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0154627Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10323 2022-12-01T10:40:43.0154786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10324 2022-12-01T10:40:43.0154940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10325 2022-12-01T10:40:43.0155094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10326 2022-12-01T10:40:43.0155355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0155478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0155757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0155890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0156075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3ak_o7i 2022-12-01T10:40:43.0156267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3ak_o7i/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0156540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0156662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0156928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0157111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0157295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06969xoa 2022-12-01T10:40:43.0157485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06969xoa/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0157756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0157905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0158318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0158523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0158719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__4ndjqd 2022-12-01T10:40:43.0158902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__4ndjqd/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0159184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0159308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0159589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0159726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0159913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ajfukqh 2022-12-01T10:40:43.0160111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ajfukqh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0160279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0160445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0160598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0160761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0160940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0161120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0161296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0161468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0161766Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0162064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0162355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0162633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0162703Z ok (3.244s) 2022-12-01T10:40:43.0162709Z 2022-12-01T10:40:43.0162904Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0162981Z Ran 1 test in 3.245s 2022-12-01T10:40:43.0162987Z 2022-12-01T10:40:43.0163049Z OK 2022-12-01T10:40:43.0163053Z 2022-12-01T10:40:43.0163139Z Generating XML reports... 2022-12-01T10:40:43.0163480Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103202.xml 2022-12-01T10:40:43.0163757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0163928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0164213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0164351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0164541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrhk0pox 2022-12-01T10:40:43.0164740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrhk0pox/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0165034Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0165039Z 2022-12-01T10:40:43.0165115Z Running tests... 2022-12-01T10:40:43.0165313Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0165569Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10558 2022-12-01T10:40:43.0165715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10559 2022-12-01T10:40:43.0165871Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10560 2022-12-01T10:40:43.0166025Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10561 2022-12-01T10:40:43.0166300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0166427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0166709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0166846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0167034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9ovpbar 2022-12-01T10:40:43.0167235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9ovpbar/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0167494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0167617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0167946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0168086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0168272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5ddsi8_ 2022-12-01T10:40:43.0168470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5ddsi8_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0168743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0168870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0169134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0169274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0169460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmnlwlrl 2022-12-01T10:40:43.0169658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmnlwlrl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0169932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0170056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0170336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0170472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0170690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyghkq8en 2022-12-01T10:40:43.0170873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyghkq8en/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0171039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0171203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0171393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0171552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0171730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0171907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0172084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0172384Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0172543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0172838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0173139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0173427Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0173496Z ok (3.140s) 2022-12-01T10:40:43.0173501Z 2022-12-01T10:40:43.0173699Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0173778Z Ran 1 test in 3.140s 2022-12-01T10:40:43.0173783Z 2022-12-01T10:40:43.0173845Z OK 2022-12-01T10:40:43.0173850Z 2022-12-01T10:40:43.0173921Z Generating XML reports... 2022-12-01T10:40:43.0174259Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103206.xml 2022-12-01T10:40:43.0174534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0174658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0174938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0175071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0175256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcu4lajl 2022-12-01T10:40:43.0175455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcu4lajl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0175709Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0175714Z 2022-12-01T10:40:43.0175774Z Running tests... 2022-12-01T10:40:43.0175967Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0176210Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10793 2022-12-01T10:40:43.0176369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10794 2022-12-01T10:40:43.0176523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10795 2022-12-01T10:40:43.0176674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10796 2022-12-01T10:40:43.0176947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0177100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0177385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0177507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0177695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsup3bin8 2022-12-01T10:40:43.0177920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsup3bin8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0178195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0178317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0178597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0178733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0178917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdaal3avc 2022-12-01T10:40:43.0179100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdaal3avc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0179371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0179495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0179774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0179906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0180090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwr056t1 2022-12-01T10:40:43.0180283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwr056t1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0180553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0180676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0180945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0181076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0181260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyomyexew 2022-12-01T10:40:43.0181455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyomyexew/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0181616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0181777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0181938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0182102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0182157Z ok (2.143s) 2022-12-01T10:40:43.0182173Z 2022-12-01T10:40:43.0182358Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0182432Z Ran 1 test in 2.143s 2022-12-01T10:40:43.0182437Z 2022-12-01T10:40:43.0182499Z OK 2022-12-01T10:40:43.0182504Z 2022-12-01T10:40:43.0182587Z Generating XML reports... 2022-12-01T10:40:43.0182925Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103211.xml 2022-12-01T10:40:43.0183197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0183320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0183633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0183756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0183939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7e6gv6i 2022-12-01T10:40:43.0184133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7e6gv6i/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0184417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0184422Z 2022-12-01T10:40:43.0184494Z Running tests... 2022-12-01T10:40:43.0184688Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0184907Z test_rpc_return_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11017 2022-12-01T10:40:43.0185064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11018 2022-12-01T10:40:43.0185209Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11019 2022-12-01T10:40:43.0185361Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11020 2022-12-01T10:40:43.0185636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0185759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0186042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0186179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0186362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpim6ld05g 2022-12-01T10:40:43.0186557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpim6ld05g/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0186831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0186941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0187219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0187352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0187539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7pvsun8 2022-12-01T10:40:43.0187736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7pvsun8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0188007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0188130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0188415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0188549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0188720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprokl93p7 2022-12-01T10:40:43.0188911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprokl93p7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0189186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0189308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0189587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0189719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0189900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsg527w02 2022-12-01T10:40:43.0190121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsg527w02/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0190272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0190435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0190596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0190790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0190857Z ok (2.138s) 2022-12-01T10:40:43.0190862Z 2022-12-01T10:40:43.0191061Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0191139Z Ran 1 test in 2.138s 2022-12-01T10:40:43.0191144Z 2022-12-01T10:40:43.0191204Z OK 2022-12-01T10:40:43.0191208Z 2022-12-01T10:40:43.0191279Z Generating XML reports... 2022-12-01T10:40:43.0191622Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103215.xml 2022-12-01T10:40:43.0191895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0192019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0192295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0192432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0192615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nzzjhr4 2022-12-01T10:40:43.0192810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nzzjhr4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0193067Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0193072Z 2022-12-01T10:40:43.0193131Z Running tests... 2022-12-01T10:40:43.0193330Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0193546Z test_rpc_timeouts (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11240 2022-12-01T10:40:43.0193702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11241 2022-12-01T10:40:43.0193856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11242 2022-12-01T10:40:43.0194010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11243 2022-12-01T10:40:43.0194287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0194408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0194688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0194812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0194996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9n1xbfv 2022-12-01T10:40:43.0195190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9n1xbfv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0195460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0195585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0195866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0196000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0196183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9yxqlsy 2022-12-01T10:40:43.0196396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9yxqlsy/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0196668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0196789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0197066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0197199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0197409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthchtrwj 2022-12-01T10:40:43.0197604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthchtrwj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0197873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0197995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0198410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0198547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0198731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpla4e_lb7 2022-12-01T10:40:43.0198924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpla4e_lb7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0199090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0199257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0199418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0199583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0199637Z ok (8.251s) 2022-12-01T10:40:43.0199656Z 2022-12-01T10:40:43.0199846Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0199925Z Ran 1 test in 8.251s 2022-12-01T10:40:43.0199931Z 2022-12-01T10:40:43.0199993Z OK 2022-12-01T10:40:43.0199998Z 2022-12-01T10:40:43.0200083Z Generating XML reports... 2022-12-01T10:40:43.0200424Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103218.xml 2022-12-01T10:40:43.0200703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0200827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0201110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0201231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0201417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdjpcscx 2022-12-01T10:40:43.0201615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdjpcscx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0201872Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0201877Z 2022-12-01T10:40:43.0201949Z Running tests... 2022-12-01T10:40:43.0202144Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0202373Z test_rref_context_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11463 2022-12-01T10:40:43.0202531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11464 2022-12-01T10:40:43.0202672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11465 2022-12-01T10:40:43.0202825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11466 2022-12-01T10:40:43.0203149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0203274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0203553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0203687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0203907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmt4br1w 2022-12-01T10:40:43.0204103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmt4br1w/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0204377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0204486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0204770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0204904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0205090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzm9ma9q 2022-12-01T10:40:43.0205285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzm9ma9q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0205555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0205680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0205959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0206094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0206264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw1krhjd 2022-12-01T10:40:43.0206461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw1krhjd/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0206733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0206856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0207132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0207267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0207452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhszzlur 2022-12-01T10:40:43.0207647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhszzlur/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0207852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0208023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0208187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0208350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0208527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0208707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0208884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0209057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0209359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0209683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0209978Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0210271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0210340Z ok (2.345s) 2022-12-01T10:40:43.0210373Z 2022-12-01T10:40:43.0210572Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0210647Z Ran 1 test in 2.345s 2022-12-01T10:40:43.0210652Z 2022-12-01T10:40:43.0210713Z OK 2022-12-01T10:40:43.0210718Z 2022-12-01T10:40:43.0210801Z Generating XML reports... 2022-12-01T10:40:43.0211145Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103228.xml 2022-12-01T10:40:43.0211410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0211533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0211812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0211946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0212130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8x1ovjj0 2022-12-01T10:40:43.0212331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8x1ovjj0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0212588Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0212593Z 2022-12-01T10:40:43.0212667Z Running tests... 2022-12-01T10:40:43.0212849Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0213075Z test_rref_forward_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11698 2022-12-01T10:40:43.0213236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11699 2022-12-01T10:40:43.0213390Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11700 2022-12-01T10:40:43.0213545Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11701 2022-12-01T10:40:43.0213819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0213944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0214227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0214361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0214534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7o_jbl7 2022-12-01T10:40:43.0214728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7o_jbl7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0214998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0215121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0215400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0215535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0215776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehp4zsg4 2022-12-01T10:40:43.0215973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehp4zsg4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0216266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0216387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0216665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0216799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0216980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp124m85er 2022-12-01T10:40:43.0217197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp124m85er/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0217467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0217591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0217869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0217994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0218176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2r7v42u 2022-12-01T10:40:43.0218369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2r7v42u/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0218534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0218702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0218866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0219030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0219098Z ok (2.137s) 2022-12-01T10:40:43.0219103Z 2022-12-01T10:40:43.0219303Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0219366Z Ran 1 test in 2.137s 2022-12-01T10:40:43.0219374Z 2022-12-01T10:40:43.0219436Z OK 2022-12-01T10:40:43.0219441Z 2022-12-01T10:40:43.0219526Z Generating XML reports... 2022-12-01T10:40:43.0219866Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103232.xml 2022-12-01T10:40:43.0220137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0220266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0220546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0220682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0220855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3mjyl4e 2022-12-01T10:40:43.0221054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3mjyl4e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0221312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0221317Z 2022-12-01T10:40:43.0221392Z Running tests... 2022-12-01T10:40:43.0221588Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0221812Z test_rref_get_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11921 2022-12-01T10:40:43.0221972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11922 2022-12-01T10:40:43.0222129Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11923 2022-12-01T10:40:43.0222286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11924 2022-12-01T10:40:43.0222548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0222721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0223011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0223148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0223333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoaoyaexg 2022-12-01T10:40:43.0223528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoaoyaexg/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0223837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0223960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0224226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0224362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0224548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnq_q8lay 2022-12-01T10:40:43.0224743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnq_q8lay/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0225015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0225138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0225421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0225558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0225742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zpjfayp 2022-12-01T10:40:43.0225923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zpjfayp/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0226195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0226319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0226596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0226730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0226931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvdb1x52 2022-12-01T10:40:43.0227185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvdb1x52/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0227396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0227547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0227712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0227874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0227943Z ok (2.140s) 2022-12-01T10:40:43.0227948Z 2022-12-01T10:40:43.0228150Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0228225Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0228229Z 2022-12-01T10:40:43.0228293Z OK 2022-12-01T10:40:43.0228297Z 2022-12-01T10:40:43.0228382Z Generating XML reports... 2022-12-01T10:40:43.0228720Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103235.xml 2022-12-01T10:40:43.0228979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0229103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0229430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0229567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0229750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_00avijh 2022-12-01T10:40:43.0229942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_00avijh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0230201Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0230235Z 2022-12-01T10:40:43.0230310Z Running tests... 2022-12-01T10:40:43.0230495Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0230707Z test_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12144 2022-12-01T10:40:43.0230864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12145 2022-12-01T10:40:43.0231022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12146 2022-12-01T10:40:43.0231175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12147 2022-12-01T10:40:43.0231449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0231574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0231857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0231994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0232165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsu45w0ob 2022-12-01T10:40:43.0232360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsu45w0ob/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0232632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0232756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0233033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0233167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0233350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40196kdv 2022-12-01T10:40:43.0233543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40196kdv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0233800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0233923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0234205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0234340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0234524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc56z_qvc 2022-12-01T10:40:43.0234715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc56z_qvc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0234988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0235112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0235394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0235515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0235702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6uf7pgo 2022-12-01T10:40:43.0235925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6uf7pgo/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0236087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0236250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0236410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0236571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0236777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0236943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0237116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0237289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0237591Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0237888Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0238333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0238632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0239191Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-12-01T10:40:43.0239380Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1) 2022-12-01T10:40:43.0239386Z 2022-12-01T10:40:43.0239930Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-12-01T10:40:43.0240117Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1) 2022-12-01T10:40:43.0240122Z 2022-12-01T10:40:43.0240651Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-12-01T10:40:43.0240837Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1) 2022-12-01T10:40:43.0240843Z 2022-12-01T10:40:43.0241417Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-12-01T10:40:43.0241597Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1) 2022-12-01T10:40:43.0241602Z 2022-12-01T10:40:43.0241668Z ok (2.138s) 2022-12-01T10:40:43.0241673Z 2022-12-01T10:40:43.0241873Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0241984Z Ran 1 test in 2.139s 2022-12-01T10:40:43.0241989Z 2022-12-01T10:40:43.0242037Z OK 2022-12-01T10:40:43.0242041Z 2022-12-01T10:40:43.0242126Z Generating XML reports... 2022-12-01T10:40:43.0242469Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103239.xml 2022-12-01T10:40:43.0242747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0242876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0243162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0243300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0243488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwv0mbx42 2022-12-01T10:40:43.0243672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwv0mbx42/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0243928Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0243933Z 2022-12-01T10:40:43.0244004Z Running tests... 2022-12-01T10:40:43.0244200Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0244421Z test_rref_proxy_class (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12379 2022-12-01T10:40:43.0244580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12380 2022-12-01T10:40:43.0244738Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12381 2022-12-01T10:40:43.0244891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12382 2022-12-01T10:40:43.0245165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0245278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0245560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0245695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0245880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmajng7ok 2022-12-01T10:40:43.0246078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmajng7ok/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0246349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0246471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0246751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0246884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0247057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjg6e38mi 2022-12-01T10:40:43.0247251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjg6e38mi/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0247524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0247648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0248032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0248170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0248356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptak7spoh 2022-12-01T10:40:43.0248551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptak7spoh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0248843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0248967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0249248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0249382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0249566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ilius0s 2022-12-01T10:40:43.0249761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ilius0s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0249928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0250091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0250256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0250404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0250472Z ok (2.144s) 2022-12-01T10:40:43.0250477Z 2022-12-01T10:40:43.0250675Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0250751Z Ran 1 test in 2.144s 2022-12-01T10:40:43.0250756Z 2022-12-01T10:40:43.0250818Z OK 2022-12-01T10:40:43.0250823Z 2022-12-01T10:40:43.0250910Z Generating XML reports... 2022-12-01T10:40:43.0251251Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103242.xml 2022-12-01T10:40:43.0251529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0251641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0251922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0252060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0252242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5w91ylet 2022-12-01T10:40:43.0252435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5w91ylet/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0252691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0252697Z 2022-12-01T10:40:43.0252769Z Running tests... 2022-12-01T10:40:43.0252965Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0253187Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12602 2022-12-01T10:40:43.0253331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12603 2022-12-01T10:40:43.0253489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12604 2022-12-01T10:40:43.0253642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12605 2022-12-01T10:40:43.0253912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0254035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0254343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0254478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0254662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv4q2cvd8 2022-12-01T10:40:43.0254843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv4q2cvd8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0255114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0255279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0255558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0255692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0255878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrv4eemm 2022-12-01T10:40:43.0256073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrv4eemm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0256342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0256466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0256730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0256865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0257046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26ildt_1 2022-12-01T10:40:43.0257241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26ildt_1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0257510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0257633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0257912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0258045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0258217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctku3w6u 2022-12-01T10:40:43.0258416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctku3w6u/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0258580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0258745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0258907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0259067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0259133Z ok (2.039s) 2022-12-01T10:40:43.0259138Z 2022-12-01T10:40:43.0259335Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0259409Z Ran 1 test in 2.039s 2022-12-01T10:40:43.0259414Z 2022-12-01T10:40:43.0259461Z OK 2022-12-01T10:40:43.0259466Z 2022-12-01T10:40:43.0259549Z Generating XML reports... 2022-12-01T10:40:43.0259886Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103246.xml 2022-12-01T10:40:43.0260164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0260288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0260567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0260727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0260913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cjqgtun 2022-12-01T10:40:43.0261095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cjqgtun/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0261350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0261356Z 2022-12-01T10:40:43.0261455Z Running tests... 2022-12-01T10:40:43.0261652Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0261873Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12825 2022-12-01T10:40:43.0262030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12826 2022-12-01T10:40:43.0262184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12827 2022-12-01T10:40:43.0262340Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12828 2022-12-01T10:40:43.0262612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0262722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0263003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0263139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0263323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2bcqya7 2022-12-01T10:40:43.0263516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2bcqya7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0263785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0270411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0270787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0270928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0271105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuiiq8yds 2022-12-01T10:40:43.0271306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuiiq8yds/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0271589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0271714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0271997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0272136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0272325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclutpym2 2022-12-01T10:40:43.0272521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclutpym2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0272797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0272908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0273187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0273320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0273504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__eob6vt 2022-12-01T10:40:43.0273695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__eob6vt/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0273953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0274120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0274283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0274433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0274502Z ok (2.139s) 2022-12-01T10:40:43.0274543Z 2022-12-01T10:40:43.0274747Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0274824Z Ran 1 test in 2.139s 2022-12-01T10:40:43.0274829Z 2022-12-01T10:40:43.0274892Z OK 2022-12-01T10:40:43.0274897Z 2022-12-01T10:40:43.0274981Z Generating XML reports... 2022-12-01T10:40:43.0275322Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103249.xml 2022-12-01T10:40:43.0275598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0275722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0275990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0276124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0276307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkizpvng 2022-12-01T10:40:43.0276507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkizpvng/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0276763Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0276768Z 2022-12-01T10:40:43.0276842Z Running tests... 2022-12-01T10:40:43.0277041Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0277261Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13048 2022-12-01T10:40:43.0277405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13049 2022-12-01T10:40:43.0277560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13050 2022-12-01T10:40:43.0277713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13051 2022-12-01T10:40:43.0277990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0278112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0278781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0279000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0279302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5s9o99j 2022-12-01T10:40:43.0279679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5s9o99j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0280154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0280353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0280828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0281023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0281280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1p9u39n 2022-12-01T10:40:43.0281614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1p9u39n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0282176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0282376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0282819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0283009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0283313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrxqdmu_ 2022-12-01T10:40:43.0283681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrxqdmu_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0284162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0284360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0284840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0285064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0285389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0h0leb11 2022-12-01T10:40:43.0285693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0h0leb11/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0285963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0286250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0286536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0286809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0286917Z ok (2.241s) 2022-12-01T10:40:43.0286928Z 2022-12-01T10:40:43.0287253Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0287393Z Ran 1 test in 2.241s 2022-12-01T10:40:43.0287407Z 2022-12-01T10:40:43.0287506Z OK 2022-12-01T10:40:43.0287532Z 2022-12-01T10:40:43.0287668Z Generating XML reports... 2022-12-01T10:40:43.0288327Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103253.xml 2022-12-01T10:40:43.0288789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0288996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0289481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0289708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0290024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpkz0bp1 2022-12-01T10:40:43.0290358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpkz0bp1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0290763Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0290771Z 2022-12-01T10:40:43.0290897Z Running tests... 2022-12-01T10:40:43.0291235Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0291620Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13271 2022-12-01T10:40:43.0291907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13272 2022-12-01T10:40:43.0292182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13273 2022-12-01T10:40:43.0292454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13274 2022-12-01T10:40:43.0292951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0293245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0293757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0293994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0294320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzf5o380d 2022-12-01T10:40:43.0294671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzf5o380d/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0295213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0295426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0295931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0296176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0296497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7s3t39s 2022-12-01T10:40:43.0296848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7s3t39s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0297320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0297491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0297982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0298171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0298483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcuq0zian 2022-12-01T10:40:43.0298809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcuq0zian/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0299311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0299500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0300012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0300247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0300582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf85g2ty 2022-12-01T10:40:43.0300924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf85g2ty/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0301188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0301454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0301737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0301994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0302104Z ok (2.140s) 2022-12-01T10:40:43.0302114Z 2022-12-01T10:40:43.0302459Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0302579Z Ran 1 test in 2.141s 2022-12-01T10:40:43.0302589Z 2022-12-01T10:40:43.0302688Z OK 2022-12-01T10:40:43.0302701Z 2022-12-01T10:40:43.0302845Z Generating XML reports... 2022-12-01T10:40:43.0303456Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103257.xml 2022-12-01T10:40:43.0303959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0304175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0304762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0305003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0305340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4b2mk06 2022-12-01T10:40:43.0305658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4b2mk06/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0306119Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0306182Z 2022-12-01T10:40:43.0306304Z Running tests... 2022-12-01T10:40:43.0306652Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0307051Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13494 2022-12-01T10:40:43.0307310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13495 2022-12-01T10:40:43.0307584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13496 2022-12-01T10:40:43.0307854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13497 2022-12-01T10:40:43.0308354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0308563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0309080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0309324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0309657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvocf01vi 2022-12-01T10:40:43.0310009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvocf01vi/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0310500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0310716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0311218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0311451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0311783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk2bkgu_ 2022-12-01T10:40:43.0312138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk2bkgu_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0312632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0312848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0313348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0313593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0313917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuavrlyo4 2022-12-01T10:40:43.0314262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuavrlyo4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0314757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0314974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0315461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0315695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0316025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsb038hiq 2022-12-01T10:40:43.0316435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsb038hiq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0316724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0317016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0317295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0317591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0317753Z ok (2.139s) 2022-12-01T10:40:43.0317763Z 2022-12-01T10:40:43.0318130Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0318408Z Ran 1 test in 2.139s 2022-12-01T10:40:43.0318420Z 2022-12-01T10:40:43.0318504Z OK 2022-12-01T10:40:43.0318527Z 2022-12-01T10:40:43.0318653Z Generating XML reports... 2022-12-01T10:40:43.0319287Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103300.xml 2022-12-01T10:40:43.0319787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0319999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0320506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0320745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0321070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9do256z4 2022-12-01T10:40:43.0321407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9do256z4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0321843Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0321852Z 2022-12-01T10:40:43.0321983Z Running tests... 2022-12-01T10:40:43.0322336Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0322752Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13717 2022-12-01T10:40:43.0323039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13718 2022-12-01T10:40:43.0323319Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13719 2022-12-01T10:40:43.0323599Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13720 2022-12-01T10:40:43.0324102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0324300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0324820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0325057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0325382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hp9ul55 2022-12-01T10:40:43.0325722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hp9ul55/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0326212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0326436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0326956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0327192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0327503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrhs0n97 2022-12-01T10:40:43.0328047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrhs0n97/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0328552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0328773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0329253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0329457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0329857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mplusfk 2022-12-01T10:40:43.0330205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mplusfk/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0330714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0330916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0331435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0331669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0331990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiux5ur44 2022-12-01T10:40:43.0332335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiux5ur44/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0332632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0332921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0333205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0333481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0333595Z ok (2.136s) 2022-12-01T10:40:43.0333605Z 2022-12-01T10:40:43.0333967Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0334097Z Ran 1 test in 2.136s 2022-12-01T10:40:43.0334106Z 2022-12-01T10:40:43.0334207Z OK 2022-12-01T10:40:43.0334214Z 2022-12-01T10:40:43.0334357Z Generating XML reports... 2022-12-01T10:40:43.0334979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103304.xml 2022-12-01T10:40:43.0335489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0335707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0336201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0336435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0336763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyoq1igc 2022-12-01T10:40:43.0337108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyoq1igc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0337565Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0337574Z 2022-12-01T10:40:43.0337698Z Running tests... 2022-12-01T10:40:43.0338048Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0338428Z test_rref_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13940 2022-12-01T10:40:43.0338691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13941 2022-12-01T10:40:43.0338961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13942 2022-12-01T10:40:43.0339231Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13943 2022-12-01T10:40:43.0339806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0340014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0340505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0340741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0341074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphsav5l0w 2022-12-01T10:40:43.0341490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphsav5l0w/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0341981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0342193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0342710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0342956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0343289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzu_449mx 2022-12-01T10:40:43.0343631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzu_449mx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0344134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0344354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0344850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0345087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0345407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6q28xnjo 2022-12-01T10:40:43.0345751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6q28xnjo/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0346246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0346460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0346951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0347188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0347511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2qc7sd3 2022-12-01T10:40:43.0347838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2qc7sd3/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0348127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0348419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0348704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0348986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0349096Z ok (2.039s) 2022-12-01T10:40:43.0349107Z 2022-12-01T10:40:43.0349469Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0349601Z Ran 1 test in 2.039s 2022-12-01T10:40:43.0349610Z 2022-12-01T10:40:43.0349698Z OK 2022-12-01T10:40:43.0349720Z 2022-12-01T10:40:43.0349850Z Generating XML reports... 2022-12-01T10:40:43.0350477Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103307.xml 2022-12-01T10:40:43.0350975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0351282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0351803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0352019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0352348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhi_qecj 2022-12-01T10:40:43.0352696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhi_qecj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0353207Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0353216Z 2022-12-01T10:40:43.0353347Z Running tests... 2022-12-01T10:40:43.0353698Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0354081Z test_rref_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14163 2022-12-01T10:40:43.0354361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14164 2022-12-01T10:40:43.0354651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14165 2022-12-01T10:40:43.0354926Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14166 2022-12-01T10:40:43.0355429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0355631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0356137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0356366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0356696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fr8vz67 2022-12-01T10:40:43.0357035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fr8vz67/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0357534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0357794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0358492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0358729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0359050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tk04ag6 2022-12-01T10:40:43.0359404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tk04ag6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0359899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0360107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0360612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0360840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0361162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvqxv5i4 2022-12-01T10:40:43.0361516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvqxv5i4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0362010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0362212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0362718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0362948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0363408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc37fgn1 2022-12-01T10:40:43.0363758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc37fgn1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0364049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0364336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0364617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0364963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0365080Z ok (4.144s) 2022-12-01T10:40:43.0365091Z 2022-12-01T10:40:43.0365451Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0365584Z Ran 1 test in 4.145s 2022-12-01T10:40:43.0365594Z 2022-12-01T10:40:43.0365693Z OK 2022-12-01T10:40:43.0365701Z 2022-12-01T10:40:43.0365845Z Generating XML reports... 2022-12-01T10:40:43.0366461Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103311.xml 2022-12-01T10:40:43.0366969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0367183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0367669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0368013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0368340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_mjts_f 2022-12-01T10:40:43.0368688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_mjts_f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0369149Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0369161Z 2022-12-01T10:40:43.0369285Z Running tests... 2022-12-01T10:40:43.0369632Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0370028Z test_rref_type_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14386 2022-12-01T10:40:43.0370283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14387 2022-12-01T10:40:43.0370562Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14388 2022-12-01T10:40:43.0370833Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14389 2022-12-01T10:40:43.0371339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0371553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0372063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0372300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0372626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy3peloy_ 2022-12-01T10:40:43.0372973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy3peloy_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0373455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0373674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0374171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0374404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0374729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bdxmn19 2022-12-01T10:40:43.0375156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bdxmn19/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0375658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0375863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0376333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0376627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0376957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvki4f9q 2022-12-01T10:40:43.0377300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvki4f9q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0377809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0378031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0378544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0378786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0379125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8l5469i 2022-12-01T10:40:43.0379451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8l5469i/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0379742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0380022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0380310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0380602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0380716Z ok (2.137s) 2022-12-01T10:40:43.0380726Z 2022-12-01T10:40:43.0381073Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0381192Z Ran 1 test in 2.137s 2022-12-01T10:40:43.0381206Z 2022-12-01T10:40:43.0381289Z OK 2022-12-01T10:40:43.0381296Z 2022-12-01T10:40:43.0381435Z Generating XML reports... 2022-12-01T10:40:43.0382049Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103316.xml 2022-12-01T10:40:43.0382557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0382773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0383284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0383524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0383871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrpaar1k 2022-12-01T10:40:43.0384261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrpaar1k/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0384743Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0384754Z 2022-12-01T10:40:43.0384888Z Running tests... 2022-12-01T10:40:43.0385272Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0385714Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14609 2022-12-01T10:40:43.0386014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14610 2022-12-01T10:40:43.0386307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14611 2022-12-01T10:40:43.0386688Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14612 2022-12-01T10:40:43.0387167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0387331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0387750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0387951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0388285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2_sx10g 2022-12-01T10:40:43.0388568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2_sx10g/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0388983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0389185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0389692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0389919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0390221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspnv9zbg 2022-12-01T10:40:43.0390552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspnv9zbg/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0391028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0391232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0391769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0392011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0392366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibury73e 2022-12-01T10:40:43.0392742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibury73e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0393200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0393389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0393899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0394135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0394453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ogdetks 2022-12-01T10:40:43.0394795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ogdetks/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0395076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0395369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0395657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0395935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0396047Z ok (2.139s) 2022-12-01T10:40:43.0396062Z 2022-12-01T10:40:43.0396422Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0396554Z Ran 1 test in 2.139s 2022-12-01T10:40:43.0396563Z 2022-12-01T10:40:43.0396666Z OK 2022-12-01T10:40:43.0396674Z 2022-12-01T10:40:43.0396821Z Generating XML reports... 2022-12-01T10:40:43.0397426Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103320.xml 2022-12-01T10:40:43.0397985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0398608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0399131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0399368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0399693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncy551i0 2022-12-01T10:40:43.0400135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncy551i0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0400609Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0400619Z 2022-12-01T10:40:43.0400740Z Running tests... 2022-12-01T10:40:43.0401090Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0401491Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14832 2022-12-01T10:40:43.0401750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14833 2022-12-01T10:40:43.0402029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14834 2022-12-01T10:40:43.0402303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14835 2022-12-01T10:40:43.0402808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0403027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0403536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0403773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0404102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1bjoa9s 2022-12-01T10:40:43.0404457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1bjoa9s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0404944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0405149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0405659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0405903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0406240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk11_ry4 2022-12-01T10:40:43.0406577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk11_ry4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0407074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0407288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0407851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0408085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0408405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu426jibw 2022-12-01T10:40:43.0408682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu426jibw/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0409096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0409281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0409729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0410053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0410367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7g_lpyfr 2022-12-01T10:40:43.0410696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7g_lpyfr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0410981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0411266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0411621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0411911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0412023Z ok (2.138s) 2022-12-01T10:40:43.0412035Z 2022-12-01T10:40:43.0412394Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0412519Z Ran 1 test in 2.138s 2022-12-01T10:40:43.0412532Z 2022-12-01T10:40:43.0412617Z OK 2022-12-01T10:40:43.0412625Z 2022-12-01T10:40:43.0412769Z Generating XML reports... 2022-12-01T10:40:43.0413384Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103324.xml 2022-12-01T10:40:43.0413892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0414105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0414611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0414856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0415177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3gph_2y 2022-12-01T10:40:43.0415519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3gph_2y/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0415952Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0415962Z 2022-12-01T10:40:43.0416086Z Running tests... 2022-12-01T10:40:43.0416431Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0416839Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15055 2022-12-01T10:40:43.0417124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15056 2022-12-01T10:40:43.0417405Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15057 2022-12-01T10:40:43.0417666Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15058 2022-12-01T10:40:43.0418166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0418372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0418886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0419115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0419444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsynlx88t 2022-12-01T10:40:43.0419798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsynlx88t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0420298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0420512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0421019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0421250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0421656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wo145mx 2022-12-01T10:40:43.0422000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wo145mx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0422481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0422689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0423263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0423499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0423830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzeau0ad 2022-12-01T10:40:43.0424189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzeau0ad/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0424701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0424909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0425435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0425675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0426017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmthz27y 2022-12-01T10:40:43.0426352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmthz27y/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0426627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0426892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0427157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0427406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0427512Z ok (2.140s) 2022-12-01T10:40:43.0427523Z 2022-12-01T10:40:43.0427861Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0427982Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0427992Z 2022-12-01T10:40:43.0428085Z OK 2022-12-01T10:40:43.0428092Z 2022-12-01T10:40:43.0428241Z Generating XML reports... 2022-12-01T10:40:43.0428800Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103327.xml 2022-12-01T10:40:43.0429290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0429493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0429981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0430225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0430546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25zxo41l 2022-12-01T10:40:43.0430892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25zxo41l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0431360Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0431374Z 2022-12-01T10:40:43.0431490Z Running tests... 2022-12-01T10:40:43.0431847Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0432241Z test_rref_type_slow_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15278 2022-12-01T10:40:43.0432503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15279 2022-12-01T10:40:43.0432907Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15280 2022-12-01T10:40:43.0433188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15281 2022-12-01T10:40:43.0433677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0433900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0434411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0434701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0435026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4dx1kzo 2022-12-01T10:40:43.0435370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4dx1kzo/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0435847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0436056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0436564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0436800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0437114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5l9txv89 2022-12-01T10:40:43.0437457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5l9txv89/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0437953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0438343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0438851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0439089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0439416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaiv8kg43 2022-12-01T10:40:43.0439765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaiv8kg43/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0440259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0440483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0440999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0441226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0441539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe4udgkd 2022-12-01T10:40:43.0441860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe4udgkd/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0442128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0442409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0442695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0442981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0443093Z ok (3.142s) 2022-12-01T10:40:43.0443104Z 2022-12-01T10:40:43.0443458Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0443585Z Ran 1 test in 3.142s 2022-12-01T10:40:43.0443593Z 2022-12-01T10:40:43.0443679Z OK 2022-12-01T10:40:43.0443687Z 2022-12-01T10:40:43.0443829Z Generating XML reports... 2022-12-01T10:40:43.0444590Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103331.xml 2022-12-01T10:40:43.0445085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0445262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0445704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0445911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0446288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7g7b44w 2022-12-01T10:40:43.0446597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7g7b44w/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0447025Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0447034Z 2022-12-01T10:40:43.0447152Z Running tests... 2022-12-01T10:40:43.0447510Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0448026Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15501 2022-12-01T10:40:43.0448319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15502 2022-12-01T10:40:43.0448572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15503 2022-12-01T10:40:43.0448833Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15504 2022-12-01T10:40:43.0449343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0449545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0450052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0450296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0450623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmztf0ak 2022-12-01T10:40:43.0450969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmztf0ak/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0451468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0451685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0452209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0452458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0452783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxyq7gt1 2022-12-01T10:40:43.0453133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxyq7gt1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0453642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0453856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0454366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0454609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0454961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjh5osv9 2022-12-01T10:40:43.0455324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjh5osv9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0455848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0456059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0456664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0456908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0457243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1pm6h7e 2022-12-01T10:40:43.0457601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1pm6h7e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0457946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0458238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0458534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0458817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0458980Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0459202Z ValueError('Expected error') 2022-12-01T10:40:43.0459364Z Traceback (most recent call last): 2022-12-01T10:40:43.0459851Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0460095Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0460568Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0460710Z raise ValueError(expected_err) 2022-12-01T10:40:43.0460827Z ValueError: Expected error 2022-12-01T10:40:43.0460838Z 2022-12-01T10:40:43.0460987Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0462369Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-12-01T10:40:43.0462513Z Traceback (most recent call last): 2022-12-01T10:40:43.0462950Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0463164Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0463595Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-12-01T10:40:43.0463749Z rref_type = type(rref.local_value()) 2022-12-01T10:40:43.0464193Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:43.0464558Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:43.0464743Z ValueError: On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0464943Z ValueError('Expected error') 2022-12-01T10:40:43.0465096Z Traceback (most recent call last): 2022-12-01T10:40:43.0465561Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0465798Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0466302Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0466462Z raise ValueError(expected_err) 2022-12-01T10:40:43.0466592Z ValueError: Expected error 2022-12-01T10:40:43.0466603Z 2022-12-01T10:40:43.0466627Z 2022-12-01T10:40:43.0466762Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0466966Z ValueError('Expected error') 2022-12-01T10:40:43.0467105Z Traceback (most recent call last): 2022-12-01T10:40:43.0467528Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0467830Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0468322Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0468463Z raise ValueError(expected_err) 2022-12-01T10:40:43.0468580Z ValueError: Expected error 2022-12-01T10:40:43.0468605Z 2022-12-01T10:40:43.0468731Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0470316Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-12-01T10:40:43.0470480Z Traceback (most recent call last): 2022-12-01T10:40:43.0470960Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0471187Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0471640Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-12-01T10:40:43.0471800Z rref_type = type(rref.local_value()) 2022-12-01T10:40:43.0472271Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:43.0472680Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:43.0472845Z ValueError: On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0473052Z ValueError('Expected error') 2022-12-01T10:40:43.0473207Z Traceback (most recent call last): 2022-12-01T10:40:43.0473684Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0473908Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0474416Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0474564Z raise ValueError(expected_err) 2022-12-01T10:40:43.0474701Z ValueError: Expected error 2022-12-01T10:40:43.0474712Z 2022-12-01T10:40:43.0474725Z 2022-12-01T10:40:43.0474842Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0475049Z ValueError('Expected error') 2022-12-01T10:40:43.0475204Z Traceback (most recent call last): 2022-12-01T10:40:43.0475664Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0475892Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0476422Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0476584Z raise ValueError(expected_err) 2022-12-01T10:40:43.0476717Z ValueError: Expected error 2022-12-01T10:40:43.0476725Z 2022-12-01T10:40:43.0476851Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0478514Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-12-01T10:40:43.0478643Z Traceback (most recent call last): 2022-12-01T10:40:43.0478928Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0479169Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0479438Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-12-01T10:40:43.0479537Z rref_type = type(rref.local_value()) 2022-12-01T10:40:43.0479803Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:43.0480029Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:43.0480171Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0480292Z ValueError('Expected error') 2022-12-01T10:40:43.0480386Z Traceback (most recent call last): 2022-12-01T10:40:43.0480643Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0480777Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0481061Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0481155Z raise ValueError(expected_err) 2022-12-01T10:40:43.0481241Z ValueError: Expected error 2022-12-01T10:40:43.0481247Z 2022-12-01T10:40:43.0481252Z 2022-12-01T10:40:43.0481340Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0481445Z ValueError('Expected error') 2022-12-01T10:40:43.0481533Z Traceback (most recent call last): 2022-12-01T10:40:43.0481789Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0481924Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0482205Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0482295Z raise ValueError(expected_err) 2022-12-01T10:40:43.0482380Z ValueError: Expected error 2022-12-01T10:40:43.0482385Z 2022-12-01T10:40:43.0482463Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0483316Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-12-01T10:40:43.0483411Z Traceback (most recent call last): 2022-12-01T10:40:43.0483670Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0483803Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0484063Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-12-01T10:40:43.0484164Z rref_type = type(rref.local_value()) 2022-12-01T10:40:43.0484430Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:43.0484655Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:43.0484765Z ValueError: On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0484870Z ValueError('Expected error') 2022-12-01T10:40:43.0484961Z Traceback (most recent call last): 2022-12-01T10:40:43.0485218Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0485351Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0485634Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0485726Z raise ValueError(expected_err) 2022-12-01T10:40:43.0485810Z ValueError: Expected error 2022-12-01T10:40:43.0485815Z 2022-12-01T10:40:43.0485847Z 2022-12-01T10:40:43.0485914Z ok (2.142s) 2022-12-01T10:40:43.0485920Z 2022-12-01T10:40:43.0486108Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0486184Z Ran 1 test in 2.142s 2022-12-01T10:40:43.0486189Z 2022-12-01T10:40:43.0486249Z OK 2022-12-01T10:40:43.0486254Z 2022-12-01T10:40:43.0486338Z Generating XML reports... 2022-12-01T10:40:43.0486683Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103335.xml 2022-12-01T10:40:43.0486987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0487110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0487391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0487531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0487768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkag0b8c 2022-12-01T10:40:43.0487970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkag0b8c/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0488228Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0488234Z 2022-12-01T10:40:43.0488306Z Running tests... 2022-12-01T10:40:43.0488506Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0488740Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15724 2022-12-01T10:40:43.0488898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15725 2022-12-01T10:40:43.0489053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15726 2022-12-01T10:40:43.0489196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15727 2022-12-01T10:40:43.0489475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0489599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0489883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0490020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0490205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8a_nifpr 2022-12-01T10:40:43.0490401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8a_nifpr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0490674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0490799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0491068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0491203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0491385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30j69iw4 2022-12-01T10:40:43.0491580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30j69iw4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0491856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0491980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0492262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0492398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0492602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefuiw2qn 2022-12-01T10:40:43.0492798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefuiw2qn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0493070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0493193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0493503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0493636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0493819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cn2v7cs 2022-12-01T10:40:43.0494012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cn2v7cs/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0494178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0494329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0494492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0494651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0494741Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0494864Z ValueError('Expected error') 2022-12-01T10:40:43.0494956Z Traceback (most recent call last): 2022-12-01T10:40:43.0495218Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0495352Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0495624Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0495718Z raise ValueError(expected_err) 2022-12-01T10:40:43.0495804Z ValueError: Expected error 2022-12-01T10:40:43.0495809Z 2022-12-01T10:40:43.0495898Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0496754Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-12-01T10:40:43.0496850Z Traceback (most recent call last): 2022-12-01T10:40:43.0497113Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0497244Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0497509Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-12-01T10:40:43.0497594Z rref_type = type(rref.local_value()) 2022-12-01T10:40:43.0497861Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:43.0498083Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:43.0498194Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0498312Z ValueError('Expected error') 2022-12-01T10:40:43.0498404Z Traceback (most recent call last): 2022-12-01T10:40:43.0498661Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0498798Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0499113Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0499207Z raise ValueError(expected_err) 2022-12-01T10:40:43.0499292Z ValueError: Expected error 2022-12-01T10:40:43.0499298Z 2022-12-01T10:40:43.0499302Z 2022-12-01T10:40:43.0499391Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0499509Z ValueError('Expected error') 2022-12-01T10:40:43.0499598Z Traceback (most recent call last): 2022-12-01T10:40:43.0499853Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0500013Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0500278Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0500368Z raise ValueError(expected_err) 2022-12-01T10:40:43.0500452Z ValueError: Expected error 2022-12-01T10:40:43.0500457Z 2022-12-01T10:40:43.0500545Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0501403Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-12-01T10:40:43.0501497Z Traceback (most recent call last): 2022-12-01T10:40:43.0501754Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0501887Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0502152Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-12-01T10:40:43.0502250Z rref_type = type(rref.local_value()) 2022-12-01T10:40:43.0502505Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:43.0502729Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:43.0502836Z ValueError: On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0502954Z ValueError('Expected error') 2022-12-01T10:40:43.0503046Z Traceback (most recent call last): 2022-12-01T10:40:43.0503304Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0503439Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0503719Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0503797Z raise ValueError(expected_err) 2022-12-01T10:40:43.0503883Z ValueError: Expected error 2022-12-01T10:40:43.0503888Z 2022-12-01T10:40:43.0503892Z 2022-12-01T10:40:43.0503983Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0504103Z ValueError('Expected error') 2022-12-01T10:40:43.0504195Z Traceback (most recent call last): 2022-12-01T10:40:43.0504451Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0504582Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0504857Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0504937Z raise ValueError(expected_err) 2022-12-01T10:40:43.0505021Z ValueError: Expected error 2022-12-01T10:40:43.0505025Z 2022-12-01T10:40:43.0505113Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0506008Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-12-01T10:40:43.0506101Z Traceback (most recent call last): 2022-12-01T10:40:43.0506358Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0506523Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0506785Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-12-01T10:40:43.0506883Z rref_type = type(rref.local_value()) 2022-12-01T10:40:43.0507138Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:43.0507360Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:43.0507469Z ValueError: On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0507589Z ValueError('Expected error') 2022-12-01T10:40:43.0507681Z Traceback (most recent call last): 2022-12-01T10:40:43.0507938Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0508073Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0508355Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0508436Z raise ValueError(expected_err) 2022-12-01T10:40:43.0508521Z ValueError: Expected error 2022-12-01T10:40:43.0508526Z 2022-12-01T10:40:43.0508532Z 2022-12-01T10:40:43.0508621Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0508740Z ValueError('Expected error') 2022-12-01T10:40:43.0508831Z Traceback (most recent call last): 2022-12-01T10:40:43.0509089Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0509220Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0509498Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0509576Z raise ValueError(expected_err) 2022-12-01T10:40:43.0509660Z ValueError: Expected error 2022-12-01T10:40:43.0509664Z 2022-12-01T10:40:43.0509755Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0510615Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-12-01T10:40:43.0510708Z Traceback (most recent call last): 2022-12-01T10:40:43.0510964Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0511095Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0511361Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-12-01T10:40:43.0511462Z rref_type = type(rref.local_value()) 2022-12-01T10:40:43.0511728Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-12-01T10:40:43.0511940Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-12-01T10:40:43.0512050Z ValueError: On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0512168Z ValueError('Expected error') 2022-12-01T10:40:43.0512259Z Traceback (most recent call last): 2022-12-01T10:40:43.0512542Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0512678Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0512958Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0513049Z raise ValueError(expected_err) 2022-12-01T10:40:43.0513120Z ValueError: Expected error 2022-12-01T10:40:43.0513153Z 2022-12-01T10:40:43.0513157Z 2022-12-01T10:40:43.0513225Z ok (2.239s) 2022-12-01T10:40:43.0513230Z 2022-12-01T10:40:43.0513430Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0513506Z Ran 1 test in 2.240s 2022-12-01T10:40:43.0513511Z 2022-12-01T10:40:43.0513571Z OK 2022-12-01T10:40:43.0513575Z 2022-12-01T10:40:43.0513659Z Generating XML reports... 2022-12-01T10:40:43.0514002Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103339.xml 2022-12-01T10:40:43.0514282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0514393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0514675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0514811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0515001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrqgl5f7 2022-12-01T10:40:43.0515198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrqgl5f7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0515455Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0515460Z 2022-12-01T10:40:43.0515533Z Running tests... 2022-12-01T10:40:43.0515744Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0515958Z test_scalar_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15947 2022-12-01T10:40:43.0516104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15948 2022-12-01T10:40:43.0516301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15949 2022-12-01T10:40:43.0516454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15950 2022-12-01T10:40:43.0516734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0516857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0517142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0517281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0517466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_mwvr11 2022-12-01T10:40:43.0517663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_mwvr11/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0517923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0518047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0518521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0518661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0518848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsfcvggk 2022-12-01T10:40:43.0519044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsfcvggk/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0519386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0519511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0519781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0519917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0520139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb72ncrmj 2022-12-01T10:40:43.0520333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb72ncrmj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0520606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0520730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0521013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0521149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0521334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6s6usnp 2022-12-01T10:40:43.0521516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6s6usnp/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0521682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0521851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0522013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0522174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0522243Z ok (2.140s) 2022-12-01T10:40:43.0522249Z 2022-12-01T10:40:43.0522450Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0522527Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0522533Z 2022-12-01T10:40:43.0522582Z OK 2022-12-01T10:40:43.0522586Z 2022-12-01T10:40:43.0522671Z Generating XML reports... 2022-12-01T10:40:43.0523014Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103343.xml 2022-12-01T10:40:43.0523287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0523413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0523694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0523831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0524017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagv64hy4 2022-12-01T10:40:43.0524218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagv64hy4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0524461Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0524467Z 2022-12-01T10:40:43.0524541Z Running tests... 2022-12-01T10:40:43.0524737Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0524951Z test_self_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16170 2022-12-01T10:40:43.0525111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16171 2022-12-01T10:40:43.0525266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16172 2022-12-01T10:40:43.0525419Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16173 2022-12-01T10:40:43.0525728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0525842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0526120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0526254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0526439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0dl4ib0 2022-12-01T10:40:43.0526662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0dl4ib0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0526935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0527058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0527340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0527476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0527649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3hsvl25 2022-12-01T10:40:43.0527898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3hsvl25/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0528170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0528296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0528574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0528708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0528894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafer85iq 2022-12-01T10:40:43.0529090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafer85iq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0529351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0529473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0529751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0529888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0530069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6dqevuc 2022-12-01T10:40:43.0530263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6dqevuc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0530427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0530592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0530755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0530902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0530969Z ok (2.139s) 2022-12-01T10:40:43.0530975Z 2022-12-01T10:40:43.0531174Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0531249Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0531256Z 2022-12-01T10:40:43.0531318Z OK 2022-12-01T10:40:43.0531322Z 2022-12-01T10:40:43.0531405Z Generating XML reports... 2022-12-01T10:40:43.0531747Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103346.xml 2022-12-01T10:40:43.0532020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0532132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0532453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0532589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0532773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6naivqbz 2022-12-01T10:40:43.0532967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6naivqbz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0533249Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0533255Z 2022-12-01T10:40:43.0533328Z Running tests... 2022-12-01T10:40:43.0533525Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0533746Z test_self_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16393 2022-12-01T10:40:43.0533892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16394 2022-12-01T10:40:43.0534049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16395 2022-12-01T10:40:43.0534202Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16396 2022-12-01T10:40:43.0534474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0534596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0534877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0535012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0535197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexjbg6bp 2022-12-01T10:40:43.0535378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexjbg6bp/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0535652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0535774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0536054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0536188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0536373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbrj_hal 2022-12-01T10:40:43.0536569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbrj_hal/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0536841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0536963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0537231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0537364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0537547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyei3h9xe 2022-12-01T10:40:43.0537739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyei3h9xe/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0538012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0538135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0538416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0538550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0538766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ycow5yi 2022-12-01T10:40:43.0538950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ycow5yi/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0539114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0539276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0539437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0539632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0539700Z ok (2.041s) 2022-12-01T10:40:43.0539705Z 2022-12-01T10:40:43.0539906Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0539980Z Ran 1 test in 2.041s 2022-12-01T10:40:43.0539986Z 2022-12-01T10:40:43.0540035Z OK 2022-12-01T10:40:43.0540039Z 2022-12-01T10:40:43.0540122Z Generating XML reports... 2022-12-01T10:40:43.0540467Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103350.xml 2022-12-01T10:40:43.0540739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0540866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0541145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0541281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0541469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp360kyymj 2022-12-01T10:40:43.0541663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp360kyymj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0541906Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0541913Z 2022-12-01T10:40:43.0541986Z Running tests... 2022-12-01T10:40:43.0542184Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0542421Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16616 2022-12-01T10:40:43.0542578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16617 2022-12-01T10:40:43.0542734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16618 2022-12-01T10:40:43.0542886Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16619 2022-12-01T10:40:43.0543159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0543268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0543554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0543687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0543871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s96r1_x 2022-12-01T10:40:43.0544066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s96r1_x/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0544336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0544462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0544743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0544876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0545046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpof7l70yh 2022-12-01T10:40:43.0545273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpof7l70yh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0545542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0545664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0545941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0546108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0546291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdurhlbqh 2022-12-01T10:40:43.0546487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdurhlbqh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0546744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0546871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0547150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0547286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0547467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3rmo0fe 2022-12-01T10:40:43.0547658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3rmo0fe/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0547824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0547989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0548152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0548301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0548371Z ok (2.141s) 2022-12-01T10:40:43.0548376Z 2022-12-01T10:40:43.0548575Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0548650Z Ran 1 test in 2.141s 2022-12-01T10:40:43.0548655Z 2022-12-01T10:40:43.0548716Z OK 2022-12-01T10:40:43.0548720Z 2022-12-01T10:40:43.0548804Z Generating XML reports... 2022-12-01T10:40:43.0549142Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103353.xml 2022-12-01T10:40:43.0549415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0549526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0549803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0549938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0550123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_xfjm4t 2022-12-01T10:40:43.0550316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_xfjm4t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0550572Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0550578Z 2022-12-01T10:40:43.0550649Z Running tests... 2022-12-01T10:40:43.0550847Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0551078Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16839 2022-12-01T10:40:43.0551221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16840 2022-12-01T10:40:43.0551374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16841 2022-12-01T10:40:43.0551576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16842 2022-12-01T10:40:43.0551854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0551977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0552260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0552395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0552635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbj2dvpx 2022-12-01T10:40:43.0552820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbj2dvpx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0553092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0553214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0553499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0553635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0553821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp8ady3_ 2022-12-01T10:40:43.0554094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0554217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0554498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0554619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0554804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubawour2 2022-12-01T10:40:43.0555000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp8ady3_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0555190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubawour2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0555460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0555582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0555861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0555994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0556178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn263h1fy 2022-12-01T10:40:43.0556356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn263h1fy/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0556520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0556680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0556841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0557003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0557070Z ok (2.140s) 2022-12-01T10:40:43.0557075Z 2022-12-01T10:40:43.0557274Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0557349Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0557354Z 2022-12-01T10:40:43.0557402Z OK 2022-12-01T10:40:43.0557407Z 2022-12-01T10:40:43.0557491Z Generating XML reports... 2022-12-01T10:40:43.0557827Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103357.xml 2022-12-01T10:40:43.0558130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0558491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0558785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0558920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0559106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95cwfexz 2022-12-01T10:40:43.0559371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95cwfexz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0559617Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0559623Z 2022-12-01T10:40:43.0559695Z Running tests... 2022-12-01T10:40:43.0559892Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0560135Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17062 2022-12-01T10:40:43.0560291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17063 2022-12-01T10:40:43.0560449Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17064 2022-12-01T10:40:43.0560602Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17065 2022-12-01T10:40:43.0560876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0560989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0561268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0561401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0561587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyj9vo9c 2022-12-01T10:40:43.0561784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyj9vo9c/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0562055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0562180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0562458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0562597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0562769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpau7xoeou 2022-12-01T10:40:43.0562964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpau7xoeou/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0563238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0563362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0563640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0563776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0563959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpounn_v5k 2022-12-01T10:40:43.0564154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpounn_v5k/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0564412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0564536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0564811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0565001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0565187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm28sqdr9 2022-12-01T10:40:43.0565381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm28sqdr9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0565547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0565710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0565900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0566048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0566117Z ok (2.139s) 2022-12-01T10:40:43.0566122Z 2022-12-01T10:40:43.0566320Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0566396Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0566404Z 2022-12-01T10:40:43.0566466Z OK 2022-12-01T10:40:43.0566471Z 2022-12-01T10:40:43.0566554Z Generating XML reports... 2022-12-01T10:40:43.0566895Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103400.xml 2022-12-01T10:40:43.0567169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0567281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0567560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0567765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0567954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpra8sxoud 2022-12-01T10:40:43.0568153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpra8sxoud/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0568413Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0568419Z 2022-12-01T10:40:43.0568493Z Running tests... 2022-12-01T10:40:43.0568691Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0568925Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17285 2022-12-01T10:40:43.0569073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17286 2022-12-01T10:40:43.0569227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17287 2022-12-01T10:40:43.0569380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17288 2022-12-01T10:40:43.0569652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0569778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0570058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0570193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0570376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusufsia4 2022-12-01T10:40:43.0570570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusufsia4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0570830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0570952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0571230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0571363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0571611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmmfqty0 2022-12-01T10:40:43.0571807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmmfqty0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0572077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0572199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0572492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0572628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0572809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50upbapk 2022-12-01T10:40:43.0573001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50upbapk/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0573270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0573391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0573674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0573808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0573994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1y5j9eb 2022-12-01T10:40:43.0574174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1y5j9eb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0574335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0574497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0574660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0574819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0574886Z ok (2.239s) 2022-12-01T10:40:43.0574891Z 2022-12-01T10:40:43.0575088Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0575163Z Ran 1 test in 2.239s 2022-12-01T10:40:43.0575169Z 2022-12-01T10:40:43.0575217Z OK 2022-12-01T10:40:43.0575224Z 2022-12-01T10:40:43.0575308Z Generating XML reports... 2022-12-01T10:40:43.0575648Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103404.xml 2022-12-01T10:40:43.0575920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0576044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0576324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0576459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0576644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hzzq3ts 2022-12-01T10:40:43.0576839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hzzq3ts/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0577080Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0577087Z 2022-12-01T10:40:43.0577160Z Running tests... 2022-12-01T10:40:43.0577355Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0577568Z test_send_to_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17508 2022-12-01T10:40:43.0577725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17509 2022-12-01T10:40:43.0577912Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17510 2022-12-01T10:40:43.0578067Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17511 2022-12-01T10:40:43.0578341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0578450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0578732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0578895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0579084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1z3q7b_p 2022-12-01T10:40:43.0579279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1z3q7b_p/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0579556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0579679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0579959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0580094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0580265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7yk5vyb 2022-12-01T10:40:43.0580464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7yk5vyb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0580732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0580854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0581130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0581267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0581452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjj3nxf_ 2022-12-01T10:40:43.0581647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjj3nxf_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0581903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0582028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0582303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0582441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0582623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctf315f4 2022-12-01T10:40:43.0582817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctf315f4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0582981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0583143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0583304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0583452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0583522Z ok (2.142s) 2022-12-01T10:40:43.0583527Z 2022-12-01T10:40:43.0583725Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0583801Z Ran 1 test in 2.142s 2022-12-01T10:40:43.0583806Z 2022-12-01T10:40:43.0583867Z OK 2022-12-01T10:40:43.0583872Z 2022-12-01T10:40:43.0583956Z Generating XML reports... 2022-12-01T10:40:43.0584326Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103408.xml 2022-12-01T10:40:43.0584602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0584711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0584989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0585124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0585334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpup7mywgr 2022-12-01T10:40:43.0585531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpup7mywgr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0585786Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0585791Z 2022-12-01T10:40:43.0585863Z Running tests... 2022-12-01T10:40:43.0586060Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0586295Z test_server_process_global_profiler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17731 2022-12-01T10:40:43.0586440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17732 2022-12-01T10:40:43.0586593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17733 2022-12-01T10:40:43.0586747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17734 2022-12-01T10:40:43.0587022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0587144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0587426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0587563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0587747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpta44_y2e 2022-12-01T10:40:43.0587940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpta44_y2e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0588199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0588322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0588603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0588738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0588921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sxr3ffj 2022-12-01T10:40:43.0589116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sxr3ffj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0589385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0589508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0589775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0589908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0590093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7j3jbw6 2022-12-01T10:40:43.0590286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7j3jbw6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0590557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0590679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0591000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0591136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0591320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0wmwdtf 2022-12-01T10:40:43.0591503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0wmwdtf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0591701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0591863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0592022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0592183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0592255Z ok (2.139s) 2022-12-01T10:40:43.0592260Z 2022-12-01T10:40:43.0592457Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0592531Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0592536Z 2022-12-01T10:40:43.0592584Z OK 2022-12-01T10:40:43.0592588Z 2022-12-01T10:40:43.0592672Z Generating XML reports... 2022-12-01T10:40:43.0593011Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103411.xml 2022-12-01T10:40:43.0593279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0593405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0593685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0593818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0594006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3nkb6i8 2022-12-01T10:40:43.0594201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3nkb6i8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0594442Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0594448Z 2022-12-01T10:40:43.0594521Z Running tests... 2022-12-01T10:40:43.0594715Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0594954Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17954 2022-12-01T10:40:43.0595109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17955 2022-12-01T10:40:43.0595263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17956 2022-12-01T10:40:43.0595418Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17957 2022-12-01T10:40:43.0595694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0595803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0596084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0596219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0596404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrt336bd 2022-12-01T10:40:43.0596603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrt336bd/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0596872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0596996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0597319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0597456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0597628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoz2kav2o 2022-12-01T10:40:43.0597821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoz2kav2o/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0598092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0598422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0598711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0598844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0599028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1_6k9wm 2022-12-01T10:40:43.0599284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1_6k9wm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0599878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0600120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0600738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0601034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0601435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurjxshh8 2022-12-01T10:40:43.0601871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurjxshh8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0602228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0602599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0602951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0603284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0603422Z ok (2.038s) 2022-12-01T10:40:43.0603434Z 2022-12-01T10:40:43.0603851Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0604015Z Ran 1 test in 2.038s 2022-12-01T10:40:43.0604026Z 2022-12-01T10:40:43.0604152Z OK 2022-12-01T10:40:43.0604161Z 2022-12-01T10:40:43.0604341Z Generating XML reports... 2022-12-01T10:40:43.0605099Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103415.xml 2022-12-01T10:40:43.0605710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0605959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0606583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0606870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0607266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_n0qpkmv 2022-12-01T10:40:43.0607738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_n0qpkmv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0608309Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0608319Z 2022-12-01T10:40:43.0608471Z Running tests... 2022-12-01T10:40:43.0608889Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0609369Z test_shutdown_errors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18177 2022-12-01T10:40:43.0609839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18178 2022-12-01T10:40:43.0610182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18179 2022-12-01T10:40:43.0610512Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18180 2022-12-01T10:40:43.0611126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0611460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0612089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0612389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0612792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2s66hazl 2022-12-01T10:40:43.0613224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2s66hazl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0613820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0614076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0614709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0614995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0615403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4gcfexf 2022-12-01T10:40:43.0615832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4gcfexf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0616441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0616715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0617326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0617618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0618023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2dvauyk 2022-12-01T10:40:43.0618454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2dvauyk/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0619064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0619327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0619948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0620232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0620650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufq07sov 2022-12-01T10:40:43.0621060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufq07sov/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0621424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0621786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0622153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0622510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0622898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0623282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0623723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0624085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0624753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0625414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0626128Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0626778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0627412Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #3 to worker0, as the agent is shutting down 2022-12-01T10:40:43.0628052Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #4 to worker0, as the agent is shutting down 2022-12-01T10:40:43.0628679Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #5 to worker0, as the agent is shutting down 2022-12-01T10:40:43.0629279Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.0629878Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.0630456Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.0631018Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.0631605Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.0632181Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.0632756Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.0633344Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.0633933Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.0635130Z ERROR:torch.distributed.rpc.api:Failed to respond to 'Shutdown Proceed' in time, got error Followers ['worker1', 'worker3', 'worker2'] timed out in _all_gather after 0.00 seconds. The first exception is eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.0635332Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0635593Z RuntimeError('simulation') 2022-12-01T10:40:43.0635794Z Traceback (most recent call last): 2022-12-01T10:40:43.0636373Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0636643Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0637351Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-12-01T10:40:43.0637637Z raise RuntimeError('simulation') 2022-12-01T10:40:43.0637816Z RuntimeError: simulation 2022-12-01T10:40:43.0637826Z 2022-12-01T10:40:43.0638022Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0638371Z RuntimeError('simulation') 2022-12-01T10:40:43.0638561Z Traceback (most recent call last): 2022-12-01T10:40:43.0639151Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0639510Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0640142Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-12-01T10:40:43.0640420Z raise RuntimeError('simulation') 2022-12-01T10:40:43.0640591Z RuntimeError: simulation 2022-12-01T10:40:43.0640603Z 2022-12-01T10:40:43.0640795Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0641041Z RuntimeError('simulation') 2022-12-01T10:40:43.0641240Z Traceback (most recent call last): 2022-12-01T10:40:43.0641806Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0642070Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0642700Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-12-01T10:40:43.0642986Z raise RuntimeError('simulation') 2022-12-01T10:40:43.0643158Z RuntimeError: simulation 2022-12-01T10:40:43.0643170Z 2022-12-01T10:40:43.0643305Z ok (2.140s) 2022-12-01T10:40:43.0643314Z 2022-12-01T10:40:43.0643743Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0643898Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0643908Z 2022-12-01T10:40:43.0644032Z OK 2022-12-01T10:40:43.0644041Z 2022-12-01T10:40:43.0644200Z Generating XML reports... 2022-12-01T10:40:43.0644963Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103418.xml 2022-12-01T10:40:43.0645572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0645838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0646474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0646777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0647180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1x91dv9 2022-12-01T10:40:43.0647604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1x91dv9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0648238Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0648252Z 2022-12-01T10:40:43.0648392Z Running tests... 2022-12-01T10:40:43.0648818Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0649318Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18412 2022-12-01T10:40:43.0649659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18413 2022-12-01T10:40:43.0649997Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18414 2022-12-01T10:40:43.0650346Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18415 2022-12-01T10:40:43.0650951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0651211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0651817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0652184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0652589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cda45nw 2022-12-01T10:40:43.0653014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cda45nw/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0653623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0653927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0654558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0654848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0655247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67moq0vv 2022-12-01T10:40:43.0655655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67moq0vv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0656262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0656524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0657148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0657436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0657841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65s_nyfs 2022-12-01T10:40:43.0658262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65s_nyfs/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0658870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0659115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0659743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0660028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0660431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovlayfr_ 2022-12-01T10:40:43.0660862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovlayfr_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0661227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0661580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0661939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0662298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0662416Z ok (2.142s) 2022-12-01T10:40:43.0662431Z 2022-12-01T10:40:43.0662858Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0663015Z Ran 1 test in 2.143s 2022-12-01T10:40:43.0663026Z 2022-12-01T10:40:43.0663156Z OK 2022-12-01T10:40:43.0663166Z 2022-12-01T10:40:43.0663342Z Generating XML reports... 2022-12-01T10:40:43.0664088Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103422.xml 2022-12-01T10:40:43.0664701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0664961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0665566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0665855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0666313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mk063q4 2022-12-01T10:40:43.0666750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mk063q4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0667334Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0667346Z 2022-12-01T10:40:43.0667497Z Running tests... 2022-12-01T10:40:43.0667932Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0668481Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18635 2022-12-01T10:40:43.0668827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18636 2022-12-01T10:40:43.0669152Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18637 2022-12-01T10:40:43.0669496Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18638 2022-12-01T10:40:43.0670125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0670392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0671032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0671325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0671739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx2p6u3b 2022-12-01T10:40:43.0672166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx2p6u3b/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0672772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0673021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0673655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0673945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0674348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0m5w6ol 2022-12-01T10:40:43.0674779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0m5w6ol/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0675389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0675656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0676267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0676532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0676954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lbpl6uf 2022-12-01T10:40:43.0677389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lbpl6uf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0678000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0678403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0679041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0679332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0679734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7e8cti48 2022-12-01T10:40:43.0680157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7e8cti48/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0680495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0680957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0681313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0681671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0681922Z Rank 3 finished testing 20 times in 0.05909609794616699 seconds. 2022-12-01T10:40:43.0682173Z Rank 0 finished testing 20 times in 0.07410335540771484 seconds. 2022-12-01T10:40:43.0682488Z Rank 1 finished testing 20 times in 0.07720661163330078 seconds. 2022-12-01T10:40:43.0682731Z Rank 2 finished testing 20 times in 0.07982063293457031 seconds. 2022-12-01T10:40:43.0682851Z ok (2.138s) 2022-12-01T10:40:43.0682862Z 2022-12-01T10:40:43.0683291Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0683444Z Ran 1 test in 2.139s 2022-12-01T10:40:43.0683458Z 2022-12-01T10:40:43.0683586Z OK 2022-12-01T10:40:43.0683595Z 2022-12-01T10:40:43.0683779Z Generating XML reports... 2022-12-01T10:40:43.0684549Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103425.xml 2022-12-01T10:40:43.0685165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0685422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0686031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0686325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0686727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro3ug1dd 2022-12-01T10:40:43.0687153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro3ug1dd/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0687782Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0687796Z 2022-12-01T10:40:43.0687947Z Running tests... 2022-12-01T10:40:43.0688377Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0688894Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18858 2022-12-01T10:40:43.0689239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18859 2022-12-01T10:40:43.0689565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18860 2022-12-01T10:40:43.0689903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18861 2022-12-01T10:40:43.0690518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0690781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0691429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0691722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0692134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7zqa4rj 2022-12-01T10:40:43.0692572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7zqa4rj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0693176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0693441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0694058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0694350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0694856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw87o7bh0 2022-12-01T10:40:43.0695286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw87o7bh0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0695894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0696160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0696780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0697107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0697514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd97vf835 2022-12-01T10:40:43.0697940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd97vf835/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0698548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0698815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0699433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0699720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0700125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadj31r21 2022-12-01T10:40:43.0700538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadj31r21/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0700902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0701260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0701614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0701978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0702223Z Rank 0 finished testing 20 times in 0.11996769905090332 seconds. 2022-12-01T10:40:43.0702466Z Rank 1 finished testing 20 times in 0.11057233810424805 seconds. 2022-12-01T10:40:43.0702715Z Rank 2 finished testing 20 times in 0.11957001686096191 seconds. 2022-12-01T10:40:43.0702948Z Rank 3 finished testing 20 times in 0.11220359802246094 seconds. 2022-12-01T10:40:43.0703075Z ok (2.244s) 2022-12-01T10:40:43.0703084Z 2022-12-01T10:40:43.0703520Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0703686Z Ran 1 test in 2.244s 2022-12-01T10:40:43.0703695Z 2022-12-01T10:40:43.0703824Z OK 2022-12-01T10:40:43.0703834Z 2022-12-01T10:40:43.0704013Z Generating XML reports... 2022-12-01T10:40:43.0704749Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103429.xml 2022-12-01T10:40:43.0705353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0705613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0706200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0706491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0706902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65jclb0z 2022-12-01T10:40:43.0707323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65jclb0z/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0707886Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0707898Z 2022-12-01T10:40:43.0708050Z Running tests... 2022-12-01T10:40:43.0708473Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0709052Z test_stress_light_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19081 2022-12-01T10:40:43.0709393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19082 2022-12-01T10:40:43.0709712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19083 2022-12-01T10:40:43.0710045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19084 2022-12-01T10:40:43.0710718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0710984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0711618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0711906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0712316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl7d0wfo 2022-12-01T10:40:43.0712751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl7d0wfo/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0713333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0713600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0714222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0714516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0714926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3dzr_b0 2022-12-01T10:40:43.0715349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3dzr_b0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0715956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0716220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0716842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0717119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0717521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_wefqgr 2022-12-01T10:40:43.0717951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_wefqgr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0718632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0718900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0719535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0719829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0720230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yh9kack 2022-12-01T10:40:43.0720641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yh9kack/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0721003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0721368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0721726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0722076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0722321Z Rank 1 finished testing 1000 times in 0.25949740409851074 seconds. 2022-12-01T10:40:43.0722560Z Rank 0 finished testing 1000 times in 0.2714090347290039 seconds. 2022-12-01T10:40:43.0722954Z Rank 3 finished testing 1000 times in 0.2929213047027588 seconds. 2022-12-01T10:40:43.0723181Z Rank 2 finished testing 1000 times in 0.2679307460784912 seconds. 2022-12-01T10:40:43.0723317Z ok (2.338s) 2022-12-01T10:40:43.0723327Z 2022-12-01T10:40:43.0723757Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0723915Z Ran 1 test in 2.339s 2022-12-01T10:40:43.0723924Z 2022-12-01T10:40:43.0724053Z OK 2022-12-01T10:40:43.0724124Z 2022-12-01T10:40:43.0724305Z Generating XML reports... 2022-12-01T10:40:43.0725071Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103433.xml 2022-12-01T10:40:43.0725687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0725947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0726567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0726858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0727262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptn0hdfhb 2022-12-01T10:40:43.0727745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptn0hdfhb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0728314Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0728329Z 2022-12-01T10:40:43.0728479Z Running tests... 2022-12-01T10:40:43.0728912Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0729386Z test_use_rpc_pickler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19304 2022-12-01T10:40:43.0729713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19305 2022-12-01T10:40:43.0730058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19306 2022-12-01T10:40:43.0730395Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19307 2022-12-01T10:40:43.0730995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0731259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0731882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0732171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0732574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ezp__ej 2022-12-01T10:40:43.0732995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ezp__ej/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0733582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0733839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0734458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0734743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0735149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnn4e4sv 2022-12-01T10:40:43.0735580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnn4e4sv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0736176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0736434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0737091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0737385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0737784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjk5p7ks5 2022-12-01T10:40:43.0738205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjk5p7ks5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0738808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0739150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0739769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0740060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0740467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppul0oy3i 2022-12-01T10:40:43.0740880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppul0oy3i/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0741229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0741592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0741952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0742316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0742454Z ok (1.740s) 2022-12-01T10:40:43.0742464Z 2022-12-01T10:40:43.0742901Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0743057Z Ran 1 test in 1.740s 2022-12-01T10:40:43.0743067Z 2022-12-01T10:40:43.0743174Z OK 2022-12-01T10:40:43.0743199Z 2022-12-01T10:40:43.0743356Z Generating XML reports... 2022-12-01T10:40:43.0744114Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103436.xml 2022-12-01T10:40:43.0744719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0744984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0745611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0745905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0746303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hbjvp9s 2022-12-01T10:40:43.0746729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hbjvp9s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0747280Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0747310Z 2022-12-01T10:40:43.0747442Z Running tests... 2022-12-01T10:40:43.0747871Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0748364Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19359 2022-12-01T10:40:43.0748700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19360 2022-12-01T10:40:43.0749039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19361 2022-12-01T10:40:43.0749377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19362 2022-12-01T10:40:43.0749982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0750243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0750853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0751201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0751605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89us4cao 2022-12-01T10:40:43.0752030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89us4cao/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0752645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0752957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0753589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0753880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0754267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbknc2dnf 2022-12-01T10:40:43.0754702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbknc2dnf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0755328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0755590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0756216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0756512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0756930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4r58g_d 2022-12-01T10:40:43.0757362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4r58g_d/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0757959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0758323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0758958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0759243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0759646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvj18mmh 2022-12-01T10:40:43.0760082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvj18mmh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0760446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0760805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0761168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0761501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0761638Z ok (2.036s) 2022-12-01T10:40:43.0761648Z 2022-12-01T10:40:43.0762078Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0762234Z Ran 1 test in 2.037s 2022-12-01T10:40:43.0762244Z 2022-12-01T10:40:43.0762372Z OK 2022-12-01T10:40:43.0762380Z 2022-12-01T10:40:43.0762557Z Generating XML reports... 2022-12-01T10:40:43.0763309Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103440.xml 2022-12-01T10:40:43.0763926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0764194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0764803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0765098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0765590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21obc9jn 2022-12-01T10:40:43.0766029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21obc9jn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0766596Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0766608Z 2022-12-01T10:40:43.0766755Z Running tests... 2022-12-01T10:40:43.0767177Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0767779Z test_user_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19582 2022-12-01T10:40:43.0768107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19583 2022-12-01T10:40:43.0768447Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19584 2022-12-01T10:40:43.0768786Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19585 2022-12-01T10:40:43.0769402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0769664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0770300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0770591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0770991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86uz3cb4 2022-12-01T10:40:43.0771423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86uz3cb4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0772006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0772267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0772903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0773198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0773603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpriwg424c 2022-12-01T10:40:43.0774032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpriwg424c/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0774642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0774918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0775530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0775829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0776236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53b39asl 2022-12-01T10:40:43.0776662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53b39asl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0777279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0777545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0778177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0778478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0778888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnco5f1s7 2022-12-01T10:40:43.0779300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnco5f1s7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0779663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0780085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0780450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0780809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0780947Z ok (2.238s) 2022-12-01T10:40:43.0780960Z 2022-12-01T10:40:43.0781401Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0781598Z Ran 1 test in 2.238s 2022-12-01T10:40:43.0781610Z 2022-12-01T10:40:43.0781713Z OK 2022-12-01T10:40:43.0781740Z 2022-12-01T10:40:43.0781896Z Generating XML reports... 2022-12-01T10:40:43.0782654Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103443.xml 2022-12-01T10:40:43.0783263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0783534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0784156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0784452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0784857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ar0l6lc 2022-12-01T10:40:43.0785287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ar0l6lc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0785842Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0785870Z 2022-12-01T10:40:43.0786001Z Running tests... 2022-12-01T10:40:43.0786429Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0786918Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19825 2022-12-01T10:40:43.0787263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19826 2022-12-01T10:40:43.0787606Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19827 2022-12-01T10:40:43.0787941Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19828 2022-12-01T10:40:43.0788543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0788812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0789416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0789711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0790109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvk4giqg 2022-12-01T10:40:43.0790547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvk4giqg/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0791164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0791426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0792052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0792341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0792733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphqbktwll 2022-12-01T10:40:43.0793155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphqbktwll/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0793765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0794024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0794730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0795026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0795430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1j_pkdun 2022-12-01T10:40:43.0795852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1j_pkdun/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0796515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0796755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0797388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0797684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0798097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr2id9kb 2022-12-01T10:40:43.0798647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr2id9kb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0799006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0799359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0799725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0800062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0800201Z ok (2.139s) 2022-12-01T10:40:43.0800211Z 2022-12-01T10:40:43.0800646Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0800808Z Ran 1 test in 2.140s 2022-12-01T10:40:43.0800818Z 2022-12-01T10:40:43.0800945Z OK 2022-12-01T10:40:43.0800954Z 2022-12-01T10:40:43.0801135Z Generating XML reports... 2022-12-01T10:40:43.0801899Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103447.xml 2022-12-01T10:40:43.0802505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0802768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0803373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0803669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0804078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0ov3pa_ 2022-12-01T10:40:43.0804503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0ov3pa_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0805073Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0805083Z 2022-12-01T10:40:43.0805232Z Running tests... 2022-12-01T10:40:43.0805661Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0806174Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20048 2022-12-01T10:40:43.0806493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20049 2022-12-01T10:40:43.0806833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20050 2022-12-01T10:40:43.0807164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20051 2022-12-01T10:40:43.0807814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0808080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0808807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0809099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0809511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53n7bd0p 2022-12-01T10:40:43.0809934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53n7bd0p/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0810515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0810861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0811496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0811785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0812192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjs7divz 2022-12-01T10:40:43.0812622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjs7divz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0813230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0813491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0814099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0814392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0814797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp617pg9c9 2022-12-01T10:40:43.0815218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp617pg9c9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0815842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0816103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0816723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0817020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0817420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u335hw3 2022-12-01T10:40:43.0817823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u335hw3/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0818177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0818531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0818887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0819251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0819392Z ok (2.240s) 2022-12-01T10:40:43.0819401Z 2022-12-01T10:40:43.0819825Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0819987Z Ran 1 test in 2.240s 2022-12-01T10:40:43.0819997Z 2022-12-01T10:40:43.0820101Z OK 2022-12-01T10:40:43.0820128Z 2022-12-01T10:40:43.0820286Z Generating XML reports... 2022-12-01T10:40:43.0821039Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103450.xml 2022-12-01T10:40:43.0821655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0821919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0822539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0822879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0823281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xbhyu7f 2022-12-01T10:40:43.0823714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xbhyu7f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0824259Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0824285Z 2022-12-01T10:40:43.0824461Z Running tests... 2022-12-01T10:40:43.0824893Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0825348Z test_wait_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20271 2022-12-01T10:40:43.0825688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20272 2022-12-01T10:40:43.0826020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20273 2022-12-01T10:40:43.0826362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20274 2022-12-01T10:40:43.0826968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0827231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0827839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0828134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0828535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyc63if9 2022-12-01T10:40:43.0828963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyc63if9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0829568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0829833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0830457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0830742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0831122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lb1i1rd 2022-12-01T10:40:43.0831548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lb1i1rd/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0832151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0832408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0833031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0833319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0833729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdab2dzo9 2022-12-01T10:40:43.0834155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdab2dzo9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0834752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0834994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0835622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0835907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0836304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0syo5b4 2022-12-01T10:40:43.0836727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0syo5b4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0837140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0837500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0837854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0838288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0838434Z ok (2.047s) 2022-12-01T10:40:43.0838516Z 2022-12-01T10:40:43.0838945Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0839103Z Ran 1 test in 2.047s 2022-12-01T10:40:43.0839112Z 2022-12-01T10:40:43.0839237Z OK 2022-12-01T10:40:43.0839246Z 2022-12-01T10:40:43.0839422Z Generating XML reports... 2022-12-01T10:40:43.0840175Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103454.xml 2022-12-01T10:40:43.0840785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0841049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0841666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0841958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0842363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1tl66om 2022-12-01T10:40:43.0842794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1tl66om/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0843359Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0843369Z 2022-12-01T10:40:43.0843520Z Running tests... 2022-12-01T10:40:43.0843943Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0844447Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20494 2022-12-01T10:40:43.0844770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20495 2022-12-01T10:40:43.0845108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20496 2022-12-01T10:40:43.0845439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20497 2022-12-01T10:40:43.0846047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0846306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0846927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0847213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0847625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0ie1dqn 2022-12-01T10:40:43.0848097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0ie1dqn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0848690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0848957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0849574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0849868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0850282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj6kn676 2022-12-01T10:40:43.0850702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj6kn676/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0851373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0851641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0852246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0852548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0852970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5mnkaqb 2022-12-01T10:40:43.0853439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5mnkaqb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0854042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0854305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0854920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0855211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0855610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpli0x0bfn 2022-12-01T10:40:43.0856013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpli0x0bfn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0856364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0856725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0857080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0857430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0857813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0858198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0858580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0858954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0859604Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0860269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0860932Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0861591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0861728Z ok (2.245s) 2022-12-01T10:40:43.0861737Z 2022-12-01T10:40:43.0862168Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0862322Z Ran 1 test in 2.245s 2022-12-01T10:40:43.0862331Z 2022-12-01T10:40:43.0862461Z OK 2022-12-01T10:40:43.0862470Z 2022-12-01T10:40:43.0862645Z Generating XML reports... 2022-12-01T10:40:43.0863379Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103458.xml 2022-12-01T10:40:43.0863988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0864258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0864875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0865162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0865619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcljfd6y 2022-12-01T10:40:43.0866057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcljfd6y/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0866621Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0866630Z 2022-12-01T10:40:43.0866759Z Running tests... 2022-12-01T10:40:43.0867180Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0867727Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20729 2022-12-01T10:40:43.0868070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20730 2022-12-01T10:40:43.0868407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20731 2022-12-01T10:40:43.0868747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20732 2022-12-01T10:40:43.0869364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0869629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0870264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0870536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0870944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6lc2eqc 2022-12-01T10:40:43.0871374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6lc2eqc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0871980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0872240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0872861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0873151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0873560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmm2wfrh 2022-12-01T10:40:43.0873967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmm2wfrh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0874568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0874843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0875481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0875766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0876167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjp763vwr 2022-12-01T10:40:43.0876594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjp763vwr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0877199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0877467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0878067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0878467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0878874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ymviohv 2022-12-01T10:40:43.0879299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ymviohv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0879655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0880080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0880443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0880799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0881188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0881560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0881997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0882676Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0883052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0883727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0884390Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0885044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0885233Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0885482Z ValueError('Expected error') 2022-12-01T10:40:43.0885674Z Traceback (most recent call last): 2022-12-01T10:40:43.0886274Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0886560Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0887190Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0887391Z raise ValueError(expected_err) 2022-12-01T10:40:43.0887573Z ValueError: Expected error 2022-12-01T10:40:43.0887583Z 2022-12-01T10:40:43.0887812Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0888059Z ValueError('Expected error') 2022-12-01T10:40:43.0888253Z Traceback (most recent call last): 2022-12-01T10:40:43.0888834Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0889119Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0889744Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0889943Z raise ValueError(expected_err) 2022-12-01T10:40:43.0890130Z ValueError: Expected error 2022-12-01T10:40:43.0890141Z 2022-12-01T10:40:43.0890320Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0890552Z ValueError('Expected error') 2022-12-01T10:40:43.0890741Z Traceback (most recent call last): 2022-12-01T10:40:43.0891318Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0891600Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0892222Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0892413Z raise ValueError(expected_err) 2022-12-01T10:40:43.0892588Z ValueError: Expected error 2022-12-01T10:40:43.0892603Z 2022-12-01T10:40:43.0892790Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0893020Z ValueError('Expected error') 2022-12-01T10:40:43.0893213Z Traceback (most recent call last): 2022-12-01T10:40:43.0893781Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0894062Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0894770Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0894971Z raise ValueError(expected_err) 2022-12-01T10:40:43.0895150Z ValueError: Expected error 2022-12-01T10:40:43.0895160Z 2022-12-01T10:40:43.0895349Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0895584Z ValueError('Expected error') 2022-12-01T10:40:43.0895781Z Traceback (most recent call last): 2022-12-01T10:40:43.0896346Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0896675Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0897288Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0897489Z raise ValueError(expected_err) 2022-12-01T10:40:43.0897675Z ValueError: Expected error 2022-12-01T10:40:43.0897684Z 2022-12-01T10:40:43.0897849Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0898102Z ValueError('Expected error') 2022-12-01T10:40:43.0898291Z Traceback (most recent call last): 2022-12-01T10:40:43.0898860Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0899138Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0899772Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0899951Z raise ValueError(expected_err) 2022-12-01T10:40:43.0900126Z ValueError: Expected error 2022-12-01T10:40:43.0900135Z 2022-12-01T10:40:43.0900321Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0900576Z ValueError('Expected error') 2022-12-01T10:40:43.0900772Z Traceback (most recent call last): 2022-12-01T10:40:43.0901331Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0901617Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0902234Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0902404Z raise ValueError(expected_err) 2022-12-01T10:40:43.0902582Z ValueError: Expected error 2022-12-01T10:40:43.0902594Z 2022-12-01T10:40:43.0902776Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0903036Z ValueError('Expected error') 2022-12-01T10:40:43.0903227Z Traceback (most recent call last): 2022-12-01T10:40:43.0903797Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0904075Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0904698Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0904877Z raise ValueError(expected_err) 2022-12-01T10:40:43.0905056Z ValueError: Expected error 2022-12-01T10:40:43.0905066Z 2022-12-01T10:40:43.0905203Z ok (2.141s) 2022-12-01T10:40:43.0905214Z 2022-12-01T10:40:43.0905642Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0905798Z Ran 1 test in 2.141s 2022-12-01T10:40:43.0905809Z 2022-12-01T10:40:43.0905937Z OK 2022-12-01T10:40:43.0905948Z 2022-12-01T10:40:43.0906128Z Generating XML reports... 2022-12-01T10:40:43.0906874Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103501.xml 2022-12-01T10:40:43.0907466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0907734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0908357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0908705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0909117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwif7_b1a 2022-12-01T10:40:43.0909547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwif7_b1a/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0910121Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0910130Z 2022-12-01T10:40:43.0910335Z Running tests... 2022-12-01T10:40:43.0910767Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0911268Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20964 2022-12-01T10:40:43.0911619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20965 2022-12-01T10:40:43.0911966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20966 2022-12-01T10:40:43.0912306Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20967 2022-12-01T10:40:43.0912918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0913181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0913819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0914117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0914508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvm46wxf 2022-12-01T10:40:43.0914942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvm46wxf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0915561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0915834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0916474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0916772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0917181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5017xbio 2022-12-01T10:40:43.0917616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5017xbio/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0918354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0918604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0919228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0919525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0919926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7481sspn 2022-12-01T10:40:43.0920355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7481sspn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0920957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0921221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0921858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0922130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0922534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rrm7ke8 2022-12-01T10:40:43.0922960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rrm7ke8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0923411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0923765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0924119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0924477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0924928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.0925311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.0925670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.0926050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.0926726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0927394Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0928101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0928768Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.0928925Z ok (2.239s) 2022-12-01T10:40:43.0928934Z 2022-12-01T10:40:43.0929372Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0929532Z Ran 1 test in 2.239s 2022-12-01T10:40:43.0929544Z 2022-12-01T10:40:43.0929654Z OK 2022-12-01T10:40:43.0929662Z 2022-12-01T10:40:43.0929846Z Generating XML reports... 2022-12-01T10:40:43.0930624Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103505.xml 2022-12-01T10:40:43.0931242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0931509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0932150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0932452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0932860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6fovj_p6 2022-12-01T10:40:43.0933283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6fovj_p6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0933831Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0933843Z 2022-12-01T10:40:43.0933996Z Running tests... 2022-12-01T10:40:43.0934431Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0934930Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21199 2022-12-01T10:40:43.0935280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21200 2022-12-01T10:40:43.0935620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21201 2022-12-01T10:40:43.0935961Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21202 2022-12-01T10:40:43.0936579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0936823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0937462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0937808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0938218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpysa78cmj 2022-12-01T10:40:43.0938641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpysa78cmj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0939246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0939602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0940229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0940522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0940916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffjpjuhx 2022-12-01T10:40:43.0941357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffjpjuhx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0941962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0942233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0942860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0943150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0943561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkmxf_mh 2022-12-01T10:40:43.0943991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkmxf_mh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0944573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0944841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0945474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0945762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0946164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3x2tmuu 2022-12-01T10:40:43.0946584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3x2tmuu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0946948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0947305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0947662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0947996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0948135Z ok (2.139s) 2022-12-01T10:40:43.0948145Z 2022-12-01T10:40:43.0948578Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0948732Z Ran 1 test in 2.139s 2022-12-01T10:40:43.0948742Z 2022-12-01T10:40:43.0948877Z OK 2022-12-01T10:40:43.0948887Z 2022-12-01T10:40:43.0949067Z Generating XML reports... 2022-12-01T10:40:43.0949817Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103509.xml 2022-12-01T10:40:43.0950427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0950673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0951305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0951593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0952055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvy2e9a41 2022-12-01T10:40:43.0952491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvy2e9a41/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0953060Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0953073Z 2022-12-01T10:40:43.0953234Z Running tests... 2022-12-01T10:40:43.0953661Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0954197Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21422 2022-12-01T10:40:43.0954523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21423 2022-12-01T10:40:43.0954859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21424 2022-12-01T10:40:43.0955199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21425 2022-12-01T10:40:43.0955811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0956082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0956714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0957007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0957407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0aaatxf 2022-12-01T10:40:43.0957843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0aaatxf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0958547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0958815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0959436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0959725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0960137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvu1mkbnx 2022-12-01T10:40:43.0960564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvu1mkbnx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0961163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0961430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0962037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0962331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0962729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06er704t 2022-12-01T10:40:43.0963156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06er704t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0963771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0964036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0964668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0964962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0965360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1uwrmq0 2022-12-01T10:40:43.0965764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1uwrmq0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0966124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0966548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0966913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0967272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0967413Z ok (2.040s) 2022-12-01T10:40:43.0967424Z 2022-12-01T10:40:43.0967939Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0968166Z Ran 1 test in 2.041s 2022-12-01T10:40:43.0968176Z 2022-12-01T10:40:43.0968280Z OK 2022-12-01T10:40:43.0968292Z 2022-12-01T10:40:43.0968477Z Generating XML reports... 2022-12-01T10:40:43.0969245Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103512.xml 2022-12-01T10:40:43.0969861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0970132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0970763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0971056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0971467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47spy_qo 2022-12-01T10:40:43.0971894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47spy_qo/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0972448Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.0972458Z 2022-12-01T10:40:43.0972610Z Running tests... 2022-12-01T10:40:43.0973032Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0973529Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21645 2022-12-01T10:40:43.0973875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21646 2022-12-01T10:40:43.0974220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21647 2022-12-01T10:40:43.0974558Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21648 2022-12-01T10:40:43.0975164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0975414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0976047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0976336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0976743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vndswzg 2022-12-01T10:40:43.0977182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vndswzg/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0977790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0978058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0978674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0978969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0979363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavqnzfvm 2022-12-01T10:40:43.0979790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavqnzfvm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0980390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0980657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0981321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0981618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0982023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp606yhnud 2022-12-01T10:40:43.0982426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp606yhnud/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0983044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0983315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.0983923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.0984208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.0984616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxp0fpfs 2022-12-01T10:40:43.0985049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxp0fpfs/_remote_module_non_scriptable.py 2022-12-01T10:40:43.0985414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.0985777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.0986126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.0986457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.0986646Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.0986906Z ValueError('Expected error') 2022-12-01T10:40:43.0987115Z Traceback (most recent call last): 2022-12-01T10:40:43.0987688Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0987979Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0988609Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0988802Z raise ValueError(expected_err) 2022-12-01T10:40:43.0988962Z ValueError: Expected error 2022-12-01T10:40:43.0988971Z 2022-12-01T10:40:43.0989157Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.0989412Z ValueError('Expected error') 2022-12-01T10:40:43.0989603Z Traceback (most recent call last): 2022-12-01T10:40:43.0990168Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0990458Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0991084Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0991262Z raise ValueError(expected_err) 2022-12-01T10:40:43.0991435Z ValueError: Expected error 2022-12-01T10:40:43.0991445Z 2022-12-01T10:40:43.0991637Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.0991898Z ValueError('Expected error') 2022-12-01T10:40:43.0992099Z Traceback (most recent call last): 2022-12-01T10:40:43.0992656Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0992940Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0993587Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0993768Z raise ValueError(expected_err) 2022-12-01T10:40:43.0993948Z ValueError: Expected error 2022-12-01T10:40:43.0993958Z 2022-12-01T10:40:43.0994148Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.0994402Z ValueError('Expected error') 2022-12-01T10:40:43.0994602Z Traceback (most recent call last): 2022-12-01T10:40:43.0995245Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.0995529Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.0996156Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.0996332Z raise ValueError(expected_err) 2022-12-01T10:40:43.0996560Z ValueError: Expected error 2022-12-01T10:40:43.0996571Z 2022-12-01T10:40:43.0996708Z ok (2.142s) 2022-12-01T10:40:43.0996718Z 2022-12-01T10:40:43.0997145Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.0997304Z Ran 1 test in 2.142s 2022-12-01T10:40:43.0997313Z 2022-12-01T10:40:43.0997441Z OK 2022-12-01T10:40:43.0997451Z 2022-12-01T10:40:43.0997627Z Generating XML reports... 2022-12-01T10:40:43.0998701Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103516.xml 2022-12-01T10:40:43.0999301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.0999578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1000213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1000506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1000921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwc1zhy4w 2022-12-01T10:40:43.1001351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwc1zhy4w/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1001923Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1001932Z 2022-12-01T10:40:43.1002086Z Running tests... 2022-12-01T10:40:43.1002523Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1002982Z test_wait_all_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21868 2022-12-01T10:40:43.1003329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21869 2022-12-01T10:40:43.1003663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21870 2022-12-01T10:40:43.1004001Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21871 2022-12-01T10:40:43.1004616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1004882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1005516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1005815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1006202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcj9m3v0 2022-12-01T10:40:43.1006627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcj9m3v0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1007232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1007492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1008174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1008461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1008868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58nfprxl 2022-12-01T10:40:43.1009297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58nfprxl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1009981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1010225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1010854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1011139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1011603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zm83sg1 2022-12-01T10:40:43.1012029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zm83sg1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1012635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1012899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1013529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1013822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1014205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2852smt 2022-12-01T10:40:43.1014625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2852smt/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1014985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1015347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1015700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1016052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1016653Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1017252Z [W tensorpipe_agent.cpp:940] RPC agent for worker2 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1017889Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #3 to worker0, as the agent is shutting down 2022-12-01T10:40:43.1018506Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #1 to worker3, as the agent is shutting down 2022-12-01T10:40:43.1019133Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #1 to worker2, as the agent is shutting down 2022-12-01T10:40:43.1019756Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #1 to worker1, as the agent is shutting down 2022-12-01T10:40:43.1019902Z ok (3.038s) 2022-12-01T10:40:43.1019913Z 2022-12-01T10:40:43.1020347Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1020504Z Ran 1 test in 3.039s 2022-12-01T10:40:43.1020515Z 2022-12-01T10:40:43.1020639Z OK 2022-12-01T10:40:43.1020648Z 2022-12-01T10:40:43.1020825Z Generating XML reports... 2022-12-01T10:40:43.1021581Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103519.xml 2022-12-01T10:40:43.1022175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1022437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1023057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1023345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1023795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27jnroy9 2022-12-01T10:40:43.1024232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27jnroy9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1024802Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1024812Z 2022-12-01T10:40:43.1024964Z Running tests... 2022-12-01T10:40:43.1025393Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1025916Z test_wait_all_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22091 2022-12-01T10:40:43.1026260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22092 2022-12-01T10:40:43.1026597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22093 2022-12-01T10:40:43.1026934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22094 2022-12-01T10:40:43.1027553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1027819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1028451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1028744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1029130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdjy4c7t 2022-12-01T10:40:43.1029563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdjy4c7t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1030175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1030436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1031071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1031361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1031767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gcbex5b 2022-12-01T10:40:43.1032198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gcbex5b/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1032797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1033048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1033670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1033958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1034360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qn91e51 2022-12-01T10:40:43.1034972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1035233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1035860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1036152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1036538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jfkprk8 2022-12-01T10:40:43.1036955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qn91e51/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1037377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jfkprk8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1037735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1038140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1038602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1038964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1039160Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1039430Z ValueError('Expected error') 2022-12-01T10:40:43.1039610Z Traceback (most recent call last): 2022-12-01T10:40:43.1040264Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1040555Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1041199Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1041392Z raise ValueError(expected_err) 2022-12-01T10:40:43.1041577Z ValueError: Expected error 2022-12-01T10:40:43.1041595Z 2022-12-01T10:40:43.1041780Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1042013Z ValueError('Expected error') 2022-12-01T10:40:43.1042200Z Traceback (most recent call last): 2022-12-01T10:40:43.1042775Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1043062Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1043689Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1043887Z raise ValueError(expected_err) 2022-12-01T10:40:43.1044069Z ValueError: Expected error 2022-12-01T10:40:43.1044078Z 2022-12-01T10:40:43.1044268Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1044494Z ValueError('Expected error') 2022-12-01T10:40:43.1044688Z Traceback (most recent call last): 2022-12-01T10:40:43.1045269Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1045549Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1046162Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1046359Z raise ValueError(expected_err) 2022-12-01T10:40:43.1046538Z ValueError: Expected error 2022-12-01T10:40:43.1046548Z 2022-12-01T10:40:43.1046742Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1046976Z ValueError('Expected error') 2022-12-01T10:40:43.1047175Z Traceback (most recent call last): 2022-12-01T10:40:43.1047778Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1048069Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1048690Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1048896Z raise ValueError(expected_err) 2022-12-01T10:40:43.1049075Z ValueError: Expected error 2022-12-01T10:40:43.1049086Z 2022-12-01T10:40:43.1049269Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1049501Z ValueError('Expected error') 2022-12-01T10:40:43.1049698Z Traceback (most recent call last): 2022-12-01T10:40:43.1050265Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1050544Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1051160Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1051356Z raise ValueError(expected_err) 2022-12-01T10:40:43.1051533Z ValueError: Expected error 2022-12-01T10:40:43.1051542Z 2022-12-01T10:40:43.1051730Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1051968Z ValueError('Expected error') 2022-12-01T10:40:43.1052227Z Traceback (most recent call last): 2022-12-01T10:40:43.1052805Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1053088Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1053705Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1053892Z raise ValueError(expected_err) 2022-12-01T10:40:43.1054119Z ValueError: Expected error 2022-12-01T10:40:43.1054130Z 2022-12-01T10:40:43.1054296Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1054555Z ValueError('Expected error') 2022-12-01T10:40:43.1054748Z Traceback (most recent call last): 2022-12-01T10:40:43.1055317Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1055597Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1056221Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1056418Z raise ValueError(expected_err) 2022-12-01T10:40:43.1056600Z ValueError: Expected error 2022-12-01T10:40:43.1056609Z 2022-12-01T10:40:43.1056781Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1057040Z ValueError('Expected error') 2022-12-01T10:40:43.1057235Z Traceback (most recent call last): 2022-12-01T10:40:43.1057803Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1058088Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1058705Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1058904Z raise ValueError(expected_err) 2022-12-01T10:40:43.1059087Z ValueError: Expected error 2022-12-01T10:40:43.1059097Z 2022-12-01T10:40:43.1059273Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1059531Z ValueError('Expected error') 2022-12-01T10:40:43.1059728Z Traceback (most recent call last): 2022-12-01T10:40:43.1060291Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1060569Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1061192Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1061396Z raise ValueError(expected_err) 2022-12-01T10:40:43.1061556Z ValueError: Expected error 2022-12-01T10:40:43.1061582Z 2022-12-01T10:40:43.1061748Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1062001Z ValueError('Expected error') 2022-12-01T10:40:43.1062194Z Traceback (most recent call last): 2022-12-01T10:40:43.1062755Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1063038Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1063646Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1063841Z raise ValueError(expected_err) 2022-12-01T10:40:43.1064003Z ValueError: Expected error 2022-12-01T10:40:43.1064031Z 2022-12-01T10:40:43.1064194Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1064449Z ValueError('Expected error') 2022-12-01T10:40:43.1064649Z Traceback (most recent call last): 2022-12-01T10:40:43.1065209Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1065477Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1066091Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1066328Z raise ValueError(expected_err) 2022-12-01T10:40:43.1066489Z ValueError: Expected error 2022-12-01T10:40:43.1066499Z 2022-12-01T10:40:43.1066690Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1066942Z ValueError('Expected error') 2022-12-01T10:40:43.1067133Z Traceback (most recent call last): 2022-12-01T10:40:43.1067699Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1067973Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1068645Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1068842Z raise ValueError(expected_err) 2022-12-01T10:40:43.1069003Z ValueError: Expected error 2022-12-01T10:40:43.1069012Z 2022-12-01T10:40:43.1069199Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1069450Z ValueError('Expected error') 2022-12-01T10:40:43.1069644Z Traceback (most recent call last): 2022-12-01T10:40:43.1070228Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1070506Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1071134Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1071333Z raise ValueError(expected_err) 2022-12-01T10:40:43.1071501Z ValueError: Expected error 2022-12-01T10:40:43.1071510Z 2022-12-01T10:40:43.1071701Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1071955Z ValueError('Expected error') 2022-12-01T10:40:43.1072145Z Traceback (most recent call last): 2022-12-01T10:40:43.1072707Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1072983Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1073602Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1073776Z raise ValueError(expected_err) 2022-12-01T10:40:43.1073952Z ValueError: Expected error 2022-12-01T10:40:43.1073963Z 2022-12-01T10:40:43.1074157Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1074412Z ValueError('Expected error') 2022-12-01T10:40:43.1074606Z Traceback (most recent call last): 2022-12-01T10:40:43.1075185Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1075477Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1076115Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1076293Z raise ValueError(expected_err) 2022-12-01T10:40:43.1076478Z ValueError: Expected error 2022-12-01T10:40:43.1076489Z 2022-12-01T10:40:43.1076674Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1076933Z ValueError('Expected error') 2022-12-01T10:40:43.1077123Z Traceback (most recent call last): 2022-12-01T10:40:43.1077696Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1077980Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1078721Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1078906Z raise ValueError(expected_err) 2022-12-01T10:40:43.1079084Z ValueError: Expected error 2022-12-01T10:40:43.1079093Z 2022-12-01T10:40:43.1079285Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1079540Z ValueError('Expected error') 2022-12-01T10:40:43.1079737Z Traceback (most recent call last): 2022-12-01T10:40:43.1080318Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1080665Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1081298Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1081478Z raise ValueError(expected_err) 2022-12-01T10:40:43.1081655Z ValueError: Expected error 2022-12-01T10:40:43.1081664Z 2022-12-01T10:40:43.1081855Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1082109Z ValueError('Expected error') 2022-12-01T10:40:43.1082374Z Traceback (most recent call last): 2022-12-01T10:40:43.1082941Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1083224Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1083824Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1084014Z raise ValueError(expected_err) 2022-12-01T10:40:43.1084199Z ValueError: Expected error 2022-12-01T10:40:43.1084208Z 2022-12-01T10:40:43.1084396Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1084650Z ValueError('Expected error') 2022-12-01T10:40:43.1084848Z Traceback (most recent call last): 2022-12-01T10:40:43.1085417Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1085699Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1086303Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1086496Z raise ValueError(expected_err) 2022-12-01T10:40:43.1086678Z ValueError: Expected error 2022-12-01T10:40:43.1086687Z 2022-12-01T10:40:43.1086876Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1087128Z ValueError('Expected error') 2022-12-01T10:40:43.1087322Z Traceback (most recent call last): 2022-12-01T10:40:43.1087965Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1088247Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1088847Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1089044Z raise ValueError(expected_err) 2022-12-01T10:40:43.1089226Z ValueError: Expected error 2022-12-01T10:40:43.1089238Z 2022-12-01T10:40:43.1089428Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1089687Z ValueError('Expected error') 2022-12-01T10:40:43.1089887Z Traceback (most recent call last): 2022-12-01T10:40:43.1090447Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1090724Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1091327Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1091525Z raise ValueError(expected_err) 2022-12-01T10:40:43.1091701Z ValueError: Expected error 2022-12-01T10:40:43.1091710Z 2022-12-01T10:40:43.1091898Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1092159Z ValueError('Expected error') 2022-12-01T10:40:43.1092358Z Traceback (most recent call last): 2022-12-01T10:40:43.1092919Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1093182Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1093795Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1093985Z raise ValueError(expected_err) 2022-12-01T10:40:43.1094167Z ValueError: Expected error 2022-12-01T10:40:43.1094176Z 2022-12-01T10:40:43.1094363Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1094724Z ValueError('Expected error') 2022-12-01T10:40:43.1094925Z Traceback (most recent call last): 2022-12-01T10:40:43.1095492Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1095750Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1096377Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1096622Z raise ValueError(expected_err) 2022-12-01T10:40:43.1096807Z ValueError: Expected error 2022-12-01T10:40:43.1096817Z 2022-12-01T10:40:43.1097011Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1097267Z ValueError('Expected error') 2022-12-01T10:40:43.1097464Z Traceback (most recent call last): 2022-12-01T10:40:43.1098045Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1098314Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1098944Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1099142Z raise ValueError(expected_err) 2022-12-01T10:40:43.1099324Z ValueError: Expected error 2022-12-01T10:40:43.1099335Z 2022-12-01T10:40:43.1099525Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1099784Z ValueError('Expected error') 2022-12-01T10:40:43.1099987Z Traceback (most recent call last): 2022-12-01T10:40:43.1100558Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1100826Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1101454Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1101646Z raise ValueError(expected_err) 2022-12-01T10:40:43.1101826Z ValueError: Expected error 2022-12-01T10:40:43.1101841Z 2022-12-01T10:40:43.1102032Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1102283Z ValueError('Expected error') 2022-12-01T10:40:43.1102473Z Traceback (most recent call last): 2022-12-01T10:40:43.1103028Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1103311Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1103933Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1104128Z raise ValueError(expected_err) 2022-12-01T10:40:43.1104309Z ValueError: Expected error 2022-12-01T10:40:43.1104318Z 2022-12-01T10:40:43.1104508Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1104762Z ValueError('Expected error') 2022-12-01T10:40:43.1104952Z Traceback (most recent call last): 2022-12-01T10:40:43.1105502Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1105785Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1106415Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1106613Z raise ValueError(expected_err) 2022-12-01T10:40:43.1106796Z ValueError: Expected error 2022-12-01T10:40:43.1106805Z 2022-12-01T10:40:43.1106997Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1118585Z ValueError('Expected error') 2022-12-01T10:40:43.1118842Z Traceback (most recent call last): 2022-12-01T10:40:43.1119502Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1119791Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1120599Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1120797Z raise ValueError(expected_err) 2022-12-01T10:40:43.1120979Z ValueError: Expected error 2022-12-01T10:40:43.1120990Z 2022-12-01T10:40:43.1121179Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1121443Z ValueError('Expected error') 2022-12-01T10:40:43.1121634Z Traceback (most recent call last): 2022-12-01T10:40:43.1122203Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1122550Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1123161Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1123347Z raise ValueError(expected_err) 2022-12-01T10:40:43.1123516Z ValueError: Expected error 2022-12-01T10:40:43.1123526Z 2022-12-01T10:40:43.1123702Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1123942Z ValueError('Expected error') 2022-12-01T10:40:43.1124129Z Traceback (most recent call last): 2022-12-01T10:40:43.1124690Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1124962Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1125569Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1125749Z raise ValueError(expected_err) 2022-12-01T10:40:43.1125932Z ValueError: Expected error 2022-12-01T10:40:43.1125941Z 2022-12-01T10:40:43.1126133Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1126392Z ValueError('Expected error') 2022-12-01T10:40:43.1126590Z Traceback (most recent call last): 2022-12-01T10:40:43.1127154Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1127413Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1128094Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1128292Z raise ValueError(expected_err) 2022-12-01T10:40:43.1128474Z ValueError: Expected error 2022-12-01T10:40:43.1128484Z 2022-12-01T10:40:43.1128678Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1128930Z ValueError('Expected error') 2022-12-01T10:40:43.1129120Z Traceback (most recent call last): 2022-12-01T10:40:43.1129697Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1129960Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1130573Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1130766Z raise ValueError(expected_err) 2022-12-01T10:40:43.1130946Z ValueError: Expected error 2022-12-01T10:40:43.1130955Z 2022-12-01T10:40:43.1131146Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1131400Z ValueError('Expected error') 2022-12-01T10:40:43.1131601Z Traceback (most recent call last): 2022-12-01T10:40:43.1132164Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1132428Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1133049Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1133246Z raise ValueError(expected_err) 2022-12-01T10:40:43.1133420Z ValueError: Expected error 2022-12-01T10:40:43.1133430Z 2022-12-01T10:40:43.1133614Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1133870Z ValueError('Expected error') 2022-12-01T10:40:43.1134069Z Traceback (most recent call last): 2022-12-01T10:40:43.1134690Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1134957Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1135584Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1135779Z raise ValueError(expected_err) 2022-12-01T10:40:43.1135958Z ValueError: Expected error 2022-12-01T10:40:43.1135969Z 2022-12-01T10:40:43.1136163Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1136474Z ValueError('Expected error') 2022-12-01T10:40:43.1136670Z Traceback (most recent call last): 2022-12-01T10:40:43.1137211Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1137504Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1138116Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1138319Z raise ValueError(expected_err) 2022-12-01T10:40:43.1138503Z ValueError: Expected error 2022-12-01T10:40:43.1138511Z 2022-12-01T10:40:43.1138702Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1138962Z ValueError('Expected error') 2022-12-01T10:40:43.1139160Z Traceback (most recent call last): 2022-12-01T10:40:43.1139711Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1140001Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1140617Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1140814Z raise ValueError(expected_err) 2022-12-01T10:40:43.1140997Z ValueError: Expected error 2022-12-01T10:40:43.1141008Z 2022-12-01T10:40:43.1141195Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1141454Z ValueError('Expected error') 2022-12-01T10:40:43.1141651Z Traceback (most recent call last): 2022-12-01T10:40:43.1142197Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1142482Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1143101Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1143295Z raise ValueError(expected_err) 2022-12-01T10:40:43.1143478Z ValueError: Expected error 2022-12-01T10:40:43.1143487Z 2022-12-01T10:40:43.1143676Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1143927Z ValueError('Expected error') 2022-12-01T10:40:43.1144121Z Traceback (most recent call last): 2022-12-01T10:40:43.1144663Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1144942Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1145569Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1145767Z raise ValueError(expected_err) 2022-12-01T10:40:43.1145948Z ValueError: Expected error 2022-12-01T10:40:43.1145957Z 2022-12-01T10:40:43.1146151Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1146404Z ValueError('Expected error') 2022-12-01T10:40:43.1146577Z Traceback (most recent call last): 2022-12-01T10:40:43.1147143Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1147431Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1148050Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1148243Z raise ValueError(expected_err) 2022-12-01T10:40:43.1148422Z ValueError: Expected error 2022-12-01T10:40:43.1148431Z 2022-12-01T10:40:43.1148671Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1148933Z ValueError('Expected error') 2022-12-01T10:40:43.1149111Z Traceback (most recent call last): 2022-12-01T10:40:43.1149680Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1149969Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1150601Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1150841Z raise ValueError(expected_err) 2022-12-01T10:40:43.1151025Z ValueError: Expected error 2022-12-01T10:40:43.1151036Z 2022-12-01T10:40:43.1151176Z ok (2.140s) 2022-12-01T10:40:43.1151188Z 2022-12-01T10:40:43.1151620Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1151759Z Ran 1 test in 2.140s 2022-12-01T10:40:43.1151794Z 2022-12-01T10:40:43.1151906Z OK 2022-12-01T10:40:43.1151916Z 2022-12-01T10:40:43.1152103Z Generating XML reports... 2022-12-01T10:40:43.1152867Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103524.xml 2022-12-01T10:40:43.1153474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1153739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1154372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1154666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1155080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6c1m3vx 2022-12-01T10:40:43.1155494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6c1m3vx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1156066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1156077Z 2022-12-01T10:40:43.1156229Z Running tests... 2022-12-01T10:40:43.1156657Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1157180Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22314 2022-12-01T10:40:43.1157524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22315 2022-12-01T10:40:43.1157872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22316 2022-12-01T10:40:43.1158324Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22317 2022-12-01T10:40:43.1158955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1159204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1159844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1160139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1160553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9vv2j19 2022-12-01T10:40:43.1160985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9vv2j19/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1161590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1161859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1162490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1162767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1163264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1vr29_2 2022-12-01T10:40:43.1163696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1vr29_2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1164312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1164578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1165211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1165570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1165974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fo8xtmw 2022-12-01T10:40:43.1166409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fo8xtmw/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1166989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1167259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1167940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1168243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1168656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpwunufq 2022-12-01T10:40:43.1169097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpwunufq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1169462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1169831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1170171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1170535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1170729Z On WorkerInfo(id=1, name=worker1): 2022-12-01T10:40:43.1170997Z ValueError('Expected error') 2022-12-01T10:40:43.1171195Z Traceback (most recent call last): 2022-12-01T10:40:43.1171778Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1172075Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1172710Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1172893Z raise ValueError(expected_err) 2022-12-01T10:40:43.1173076Z ValueError: Expected error 2022-12-01T10:40:43.1173086Z 2022-12-01T10:40:43.1173280Z On WorkerInfo(id=3, name=worker3): 2022-12-01T10:40:43.1173535Z ValueError('Expected error') 2022-12-01T10:40:43.1173736Z Traceback (most recent call last): 2022-12-01T10:40:43.1174313Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1174598Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1175222Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1175399Z raise ValueError(expected_err) 2022-12-01T10:40:43.1175578Z ValueError: Expected error 2022-12-01T10:40:43.1175593Z 2022-12-01T10:40:43.1175784Z On WorkerInfo(id=0, name=worker0): 2022-12-01T10:40:43.1176040Z ValueError('Expected error') 2022-12-01T10:40:43.1176233Z Traceback (most recent call last): 2022-12-01T10:40:43.1176807Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1177090Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1177816Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1178001Z raise ValueError(expected_err) 2022-12-01T10:40:43.1178187Z ValueError: Expected error 2022-12-01T10:40:43.1178199Z 2022-12-01T10:40:43.1178391Z On WorkerInfo(id=2, name=worker2): 2022-12-01T10:40:43.1178652Z ValueError('Expected error') 2022-12-01T10:40:43.1178844Z Traceback (most recent call last): 2022-12-01T10:40:43.1179411Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-12-01T10:40:43.1179747Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-12-01T10:40:43.1180341Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-12-01T10:40:43.1180538Z raise ValueError(expected_err) 2022-12-01T10:40:43.1180722Z ValueError: Expected error 2022-12-01T10:40:43.1180732Z 2022-12-01T10:40:43.1180869Z ok (2.039s) 2022-12-01T10:40:43.1180879Z 2022-12-01T10:40:43.1181298Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1181453Z Ran 1 test in 2.039s 2022-12-01T10:40:43.1181464Z 2022-12-01T10:40:43.1181588Z OK 2022-12-01T10:40:43.1181597Z 2022-12-01T10:40:43.1181779Z Generating XML reports... 2022-12-01T10:40:43.1182523Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103527.xml 2022-12-01T10:40:43.1183111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1183379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1184005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1184296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1184695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43z82ksm 2022-12-01T10:40:43.1185107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43z82ksm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1185674Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1185684Z 2022-12-01T10:40:43.1185840Z Running tests... 2022-12-01T10:40:43.1186233Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1186718Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22537 2022-12-01T10:40:43.1187065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22538 2022-12-01T10:40:43.1187403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22539 2022-12-01T10:40:43.1187735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22540 2022-12-01T10:40:43.1188345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1188609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1189241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1189533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1189916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv5otrns 2022-12-01T10:40:43.1190355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv5otrns/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1190962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1191227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1191918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1192212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1192625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tbcgy8u 2022-12-01T10:40:43.1193064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tbcgy8u/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1193651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1193977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1194609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1194901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1195311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwghw23a4 2022-12-01T10:40:43.1195740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwghw23a4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1196355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1196625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1197246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1197527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1197924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30w7xjs0 2022-12-01T10:40:43.1198456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30w7xjs0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1198820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1199184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1199541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1199896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1200282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1200669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1201035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1201412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1202085Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1202755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1203414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1204066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1204650Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1205236Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1205807Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1206499Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1207064Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1207617Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1207855Z ok (2.439s) 2022-12-01T10:40:43.1207867Z 2022-12-01T10:40:43.1208306Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1208467Z Ran 1 test in 2.439s 2022-12-01T10:40:43.1208476Z 2022-12-01T10:40:43.1208604Z OK 2022-12-01T10:40:43.1208614Z 2022-12-01T10:40:43.1208796Z Generating XML reports... 2022-12-01T10:40:43.1209559Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103531.xml 2022-12-01T10:40:43.1210168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1210438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1211048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1211348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1211749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0it86st 2022-12-01T10:40:43.1212179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0it86st/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1212744Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1212760Z 2022-12-01T10:40:43.1212915Z Running tests... 2022-12-01T10:40:43.1213340Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1213835Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22772 2022-12-01T10:40:43.1214159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22773 2022-12-01T10:40:43.1214502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22774 2022-12-01T10:40:43.1214838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22775 2022-12-01T10:40:43.1215452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1215716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1216359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1216653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1217058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6t02izn 2022-12-01T10:40:43.1217480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6t02izn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1218067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1218336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1218960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1219264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1219664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadwgfdd2 2022-12-01T10:40:43.1220133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadwgfdd2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1220746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1221005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1221608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1221948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1222349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq3dlvzx 2022-12-01T10:40:43.1222779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq3dlvzx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1223382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1223653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1224279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1224565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1224963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp105fy1y_ 2022-12-01T10:40:43.1225363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp105fy1y_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1225724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1226082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1226440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1226800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1227193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1227583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1227971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1228331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1228993Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1229655Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1230305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1230971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1231554Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1232132Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1232708Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1233303Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1233937Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1234522Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1235110Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1235742Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #4 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1236371Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #3 to worker0, as the agent is shutting down 2022-12-01T10:40:43.1236972Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1237561Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1238139Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1238824Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #5 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1238970Z ok (3.645s) 2022-12-01T10:40:43.1238981Z 2022-12-01T10:40:43.1239412Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1239574Z Ran 1 test in 3.645s 2022-12-01T10:40:43.1239584Z 2022-12-01T10:40:43.1239714Z OK 2022-12-01T10:40:43.1239722Z 2022-12-01T10:40:43.1239900Z Generating XML reports... 2022-12-01T10:40:43.1240632Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103535.xml 2022-12-01T10:40:43.1241238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1241512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1242139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1242429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1242836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7ixzyn4 2022-12-01T10:40:43.1243271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7ixzyn4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1243844Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1243853Z 2022-12-01T10:40:43.1244003Z Running tests... 2022-12-01T10:40:43.1244408Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1244915Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23007 2022-12-01T10:40:43.1245270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23008 2022-12-01T10:40:43.1245615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23009 2022-12-01T10:40:43.1245952Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23010 2022-12-01T10:40:43.1246652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1246923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1247553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1247869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1248280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg04d3co9 2022-12-01T10:40:43.1248773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg04d3co9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1249396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1249663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1250292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1250591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1250992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvjgqrzq 2022-12-01T10:40:43.1251427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvjgqrzq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1252021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1252291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1252915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1253206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1253615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp69nwk6 2022-12-01T10:40:43.1254050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp69nwk6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1254662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1254923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1255528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1255812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1256223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfykpca06 2022-12-01T10:40:43.1256649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfykpca06/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1257000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1257353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1257716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1258070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1258464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1258824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1259498Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1259880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1260262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1260962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1261641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1262311Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1262896Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1263532Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1264104Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1264687Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1265228Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1265790Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.1265940Z ok (2.344s) 2022-12-01T10:40:43.1265950Z 2022-12-01T10:40:43.1266381Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1266538Z Ran 1 test in 2.344s 2022-12-01T10:40:43.1266549Z 2022-12-01T10:40:43.1266677Z OK 2022-12-01T10:40:43.1266686Z 2022-12-01T10:40:43.1266870Z Generating XML reports... 2022-12-01T10:40:43.1267625Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103540.xml 2022-12-01T10:40:43.1268233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1268481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1269107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1269406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1269812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_skpdbbe 2022-12-01T10:40:43.1270245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_skpdbbe/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1270810Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1270819Z 2022-12-01T10:40:43.1270973Z Running tests... 2022-12-01T10:40:43.1271399Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1271865Z test_worker_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23242 2022-12-01T10:40:43.1272184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23243 2022-12-01T10:40:43.1272529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23244 2022-12-01T10:40:43.1272869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23245 2022-12-01T10:40:43.1273482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1273747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1274375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1274716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1275133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppb1lh1lh 2022-12-01T10:40:43.1275552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppb1lh1lh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1276168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1276481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1277104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1277394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1277803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvh6mp_r 2022-12-01T10:40:43.1278321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvh6mp_r/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1278922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1279191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1279804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1280096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1280518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdamlp2nz 2022-12-01T10:40:43.1280957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdamlp2nz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1281564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1281830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1282474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1282762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1283149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91dcctly 2022-12-01T10:40:43.1283571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91dcctly/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1283944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1284310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1284672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1285034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1285172Z ok (2.138s) 2022-12-01T10:40:43.1285187Z 2022-12-01T10:40:43.1285615Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1285775Z Ran 1 test in 2.139s 2022-12-01T10:40:43.1285788Z 2022-12-01T10:40:43.1285897Z OK 2022-12-01T10:40:43.1285906Z 2022-12-01T10:40:43.1286086Z Generating XML reports... 2022-12-01T10:40:43.1286841Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103543.xml 2022-12-01T10:40:43.1287455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1287768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1288402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1288700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1289189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnrhnh2a 2022-12-01T10:40:43.1289599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnrhnh2a/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1290163Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1290172Z 2022-12-01T10:40:43.1290325Z Running tests... 2022-12-01T10:40:43.1290748Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1291296Z test_worker_info_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23465 2022-12-01T10:40:43.1291648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23466 2022-12-01T10:40:43.1291990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23467 2022-12-01T10:40:43.1292332Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23468 2022-12-01T10:40:43.1292957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1293204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1293836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1294130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1294545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3j65qiz 2022-12-01T10:40:43.1294975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3j65qiz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1295576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1295837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1296465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1296735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1297138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxku6k6ay 2022-12-01T10:40:43.1297562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxku6k6ay/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1298169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1298436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1299058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1299351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1299754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2fxkoeih 2022-12-01T10:40:43.1300178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2fxkoeih/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1300763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1301020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1301649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1301940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1302346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4vd5i4h 2022-12-01T10:40:43.1302769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4vd5i4h/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1303129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1303556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1303917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1304252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1304394Z ok (2.041s) 2022-12-01T10:40:43.1304403Z 2022-12-01T10:40:43.1304833Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1305039Z Ran 1 test in 2.042s 2022-12-01T10:40:43.1305049Z 2022-12-01T10:40:43.1305176Z OK 2022-12-01T10:40:43.1305185Z 2022-12-01T10:40:43.1305366Z Generating XML reports... 2022-12-01T10:40:43.1306120Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103547.xml 2022-12-01T10:40:43.1306724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1306976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1307603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1307894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1308294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmv0ec46k 2022-12-01T10:40:43.1308724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmv0ec46k/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1309291Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1309300Z 2022-12-01T10:40:43.1309459Z Running tests... 2022-12-01T10:40:43.1309886Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1310367Z test_world_size_one (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23688 2022-12-01T10:40:43.1310689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23689 2022-12-01T10:40:43.1311026Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23690 2022-12-01T10:40:43.1311362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23691 2022-12-01T10:40:43.1311972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1312237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1312855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1313147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1313544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q90yxox 2022-12-01T10:40:43.1313953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q90yxox/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1314559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1314823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1315439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1315734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1316137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxeaj37qj 2022-12-01T10:40:43.1316732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1316992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1317661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1317927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1318438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5hglqh4 2022-12-01T10:40:43.1318873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxeaj37qj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1319296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5hglqh4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1319985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1320253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1320880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1321160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1321554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp43tg5d5 2022-12-01T10:40:43.1321983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp43tg5d5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1322342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1322693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1323051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1323400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1323539Z ok (1.939s) 2022-12-01T10:40:43.1323548Z 2022-12-01T10:40:43.1323979Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1324138Z Ran 1 test in 1.939s 2022-12-01T10:40:43.1324147Z 2022-12-01T10:40:43.1324260Z OK 2022-12-01T10:40:43.1324270Z 2022-12-01T10:40:43.1324449Z Generating XML reports... 2022-12-01T10:40:43.1325208Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103550.xml 2022-12-01T10:40:43.1325811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1326072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1326700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1326992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1327391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_1v40jr 2022-12-01T10:40:43.1327834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_1v40jr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1328407Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1328420Z 2022-12-01T10:40:43.1328569Z Running tests... 2022-12-01T10:40:43.1328989Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1329443Z test_wrong_types (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23785 2022-12-01T10:40:43.1329777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23786 2022-12-01T10:40:43.1330119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23787 2022-12-01T10:40:43.1330453Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23788 2022-12-01T10:40:43.1331074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1331317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1332014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1332314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1332716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9n906lj 2022-12-01T10:40:43.1333141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9n906lj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1333729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1334039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1334661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1334941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1335359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn1qpj8n 2022-12-01T10:40:43.1335793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn1qpj8n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1336407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1336666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1337286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1337582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1337997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43joyn4q 2022-12-01T10:40:43.1338426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43joyn4q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1339015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1339278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1339902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1340193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1340606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvclxrbc 2022-12-01T10:40:43.1341049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvclxrbc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1341414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1341779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1342141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1342482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1342621Z ok (1.846s) 2022-12-01T10:40:43.1342633Z 2022-12-01T10:40:43.1343067Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1343227Z Ran 1 test in 1.846s 2022-12-01T10:40:43.1343237Z 2022-12-01T10:40:43.1343363Z OK 2022-12-01T10:40:43.1343373Z 2022-12-01T10:40:43.1343556Z Generating XML reports... 2022-12-01T10:40:43.1344321Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103554.xml 2022-12-01T10:40:43.1344939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1345184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1345830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1346162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1346571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6me5ze0w 2022-12-01T10:40:43.1347007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6me5ze0w/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1347584Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1347593Z 2022-12-01T10:40:43.1347789Z Running tests... 2022-12-01T10:40:43.1348217Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1348861Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23840 2022-12-01T10:40:43.1349186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23841 2022-12-01T10:40:43.1349534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23842 2022-12-01T10:40:43.1349874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23843 2022-12-01T10:40:43.1350479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1350746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1351365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1351658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1352065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nmdhtfm 2022-12-01T10:40:43.1352476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nmdhtfm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1353088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1353358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1353996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1354294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1354706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplw1p4c1f 2022-12-01T10:40:43.1355133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplw1p4c1f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1355739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1356008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1356611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1356912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1357312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfxqcha7 2022-12-01T10:40:43.1357747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfxqcha7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1358463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1358729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1359365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1359662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1360047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqsel0c9d 2022-12-01T10:40:43.1360469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqsel0c9d/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1360902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1361268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1361631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1361990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1362196Z ok (2.141s) 2022-12-01T10:40:43.1362206Z 2022-12-01T10:40:43.1362638Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1362794Z Ran 1 test in 2.142s 2022-12-01T10:40:43.1362804Z 2022-12-01T10:40:43.1362912Z OK 2022-12-01T10:40:43.1362920Z 2022-12-01T10:40:43.1363095Z Generating XML reports... 2022-12-01T10:40:43.1364044Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103557.xml 2022-12-01T10:40:43.1364658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1364927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1365556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1365847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1366255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy7kq_ps 2022-12-01T10:40:43.1366661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy7kq_ps/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1367219Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1367231Z 2022-12-01T10:40:43.1367382Z Running tests... 2022-12-01T10:40:43.1367843Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1368497Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24095 2022-12-01T10:40:43.1368845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24096 2022-12-01T10:40:43.1369178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24097 2022-12-01T10:40:43.1369528Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24098 2022-12-01T10:40:43.1370130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1370374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1370998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1371293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1371704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jnnfm8r 2022-12-01T10:40:43.1372139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jnnfm8r/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1372749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1373019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1373647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1373937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1374315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a2xo6y3 2022-12-01T10:40:43.1374731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a2xo6y3/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1375396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1375667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1376286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1376582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1377040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_m1e2zwf 2022-12-01T10:40:43.1377473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_m1e2zwf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1378060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1378328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1378960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1379241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1379644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85hygp44 2022-12-01T10:40:43.1380065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85hygp44/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1380428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1380786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1381146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1381488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1381626Z ok (2.244s) 2022-12-01T10:40:43.1381639Z 2022-12-01T10:40:43.1382062Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1382222Z Ran 1 test in 2.245s 2022-12-01T10:40:43.1382232Z 2022-12-01T10:40:43.1382357Z OK 2022-12-01T10:40:43.1382366Z 2022-12-01T10:40:43.1382541Z Generating XML reports... 2022-12-01T10:40:43.1383493Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103601.xml 2022-12-01T10:40:43.1384102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1384359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1384959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1385246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1385653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps27gv64l 2022-12-01T10:40:43.1386081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps27gv64l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1386650Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1386662Z 2022-12-01T10:40:43.1386815Z Running tests... 2022-12-01T10:40:43.1387243Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1387887Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24395 2022-12-01T10:40:43.1388200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24396 2022-12-01T10:40:43.1388539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24397 2022-12-01T10:40:43.1388874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24398 2022-12-01T10:40:43.1389522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1389792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1390423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1390723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1391214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwnehjg4 2022-12-01T10:40:43.1391647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwnehjg4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1392234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1392494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1393129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1393424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1393823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rxidq4c 2022-12-01T10:40:43.1394246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rxidq4c/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1394848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1395118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1395722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1396011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1396414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppctz_rgi 2022-12-01T10:40:43.1396843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppctz_rgi/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1397449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1397714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1398510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1398814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1399221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqb6hbsfj 2022-12-01T10:40:43.1399631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqb6hbsfj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1399991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1400358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1400715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1401072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1401216Z ok (2.240s) 2022-12-01T10:40:43.1401226Z 2022-12-01T10:40:43.1401649Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1401807Z Ran 1 test in 2.240s 2022-12-01T10:40:43.1401818Z 2022-12-01T10:40:43.1401929Z OK 2022-12-01T10:40:43.1401960Z 2022-12-01T10:40:43.1402117Z Generating XML reports... 2022-12-01T10:40:43.1403075Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103604.xml 2022-12-01T10:40:43.1403783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1404050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1404682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1404972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1405383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5lf1dey 2022-12-01T10:40:43.1405875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5lf1dey/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1406432Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1406463Z 2022-12-01T10:40:43.1406600Z Running tests... 2022-12-01T10:40:43.1407035Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1407668Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24650 2022-12-01T10:40:43.1408056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24651 2022-12-01T10:40:43.1408403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24652 2022-12-01T10:40:43.1408744Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24653 2022-12-01T10:40:43.1409365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1409638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1410256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1410554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1410969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl91042j5 2022-12-01T10:40:43.1411404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl91042j5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1412014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1412279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1412902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1413196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1413581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgocf9v3r 2022-12-01T10:40:43.1414186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1414448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1415082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1415385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1415789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_vhex19 2022-12-01T10:40:43.1416219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgocf9v3r/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1416647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_vhex19/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1417264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1417511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1418130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1418479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1418891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobk6grn1 2022-12-01T10:40:43.1419313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobk6grn1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1419673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1420033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1420439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1420778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1420919Z ok (2.142s) 2022-12-01T10:40:43.1420930Z 2022-12-01T10:40:43.1421361Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1421520Z Ran 1 test in 2.142s 2022-12-01T10:40:43.1421535Z 2022-12-01T10:40:43.1421659Z OK 2022-12-01T10:40:43.1421669Z 2022-12-01T10:40:43.1421847Z Generating XML reports... 2022-12-01T10:40:43.1422797Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103608.xml 2022-12-01T10:40:43.1423413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1423679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1424282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1424573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1424971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3tlvq78 2022-12-01T10:40:43.1425399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3tlvq78/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1425964Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1425974Z 2022-12-01T10:40:43.1426132Z Running tests... 2022-12-01T10:40:43.1426560Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1427192Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24920 2022-12-01T10:40:43.1427541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24921 2022-12-01T10:40:43.1427854Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24922 2022-12-01T10:40:43.1428196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24923 2022-12-01T10:40:43.1428804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1429073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1429695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1429991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1430396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_pntk5n 2022-12-01T10:40:43.1430834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_pntk5n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1431422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1431690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1432305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1432642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1433053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntmvhwmp 2022-12-01T10:40:43.1433487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntmvhwmp/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1434090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1434357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1435028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1435298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1435697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8kewylu 2022-12-01T10:40:43.1436128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8kewylu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1436735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1437002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1437628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1437921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1438460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4awyx_v 2022-12-01T10:40:43.1438872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4awyx_v/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1439241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1439607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1439969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1440328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1440476Z ok (2.240s) 2022-12-01T10:40:43.1440486Z 2022-12-01T10:40:43.1440923Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1441085Z Ran 1 test in 2.240s 2022-12-01T10:40:43.1441096Z 2022-12-01T10:40:43.1441228Z OK 2022-12-01T10:40:43.1441238Z 2022-12-01T10:40:43.1441399Z Generating XML reports... 2022-12-01T10:40:43.1442354Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103612.xml 2022-12-01T10:40:43.1442962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1443227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1443853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1444145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1444553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnckmw1y4 2022-12-01T10:40:43.1444980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnckmw1y4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1445532Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1445563Z 2022-12-01T10:40:43.1445698Z Running tests... 2022-12-01T10:40:43.1446123Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1446738Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25199 2022-12-01T10:40:43.1447152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25200 2022-12-01T10:40:43.1447489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25201 2022-12-01T10:40:43.1447872Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25202 2022-12-01T10:40:43.1448476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1448741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1449417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1449714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1450121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cas3y8p 2022-12-01T10:40:43.1450728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1450991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1451615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1451909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1452316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlobdt23 2022-12-01T10:40:43.1452754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cas3y8p/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1453159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlobdt23/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1453772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1454036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1454662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1454956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1455363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd8b4u3i 2022-12-01T10:40:43.1455785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd8b4u3i/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1456395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1456645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1457270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1457561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1457968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpgfdkk3 2022-12-01T10:40:43.1458396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpgfdkk3/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1458758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1459127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1459489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1459853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1459973Z ok (2.238s) 2022-12-01T10:40:43.1459985Z 2022-12-01T10:40:43.1460407Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1460564Z Ran 1 test in 2.238s 2022-12-01T10:40:43.1460573Z 2022-12-01T10:40:43.1460701Z OK 2022-12-01T10:40:43.1460709Z 2022-12-01T10:40:43.1460886Z Generating XML reports... 2022-12-01T10:40:43.1461885Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103615.xml 2022-12-01T10:40:43.1462498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1462764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1463369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1463708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1464117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40nn2xka 2022-12-01T10:40:43.1464540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40nn2xka/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1465103Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1465118Z 2022-12-01T10:40:43.1465273Z Running tests... 2022-12-01T10:40:43.1465702Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1466351Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25478 2022-12-01T10:40:43.1466702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25479 2022-12-01T10:40:43.1467023Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25480 2022-12-01T10:40:43.1467360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25481 2022-12-01T10:40:43.1467969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1468234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1468872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1469163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1469568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wdb1m3m 2022-12-01T10:40:43.1469996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wdb1m3m/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1470599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1470845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1471467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1471755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1472164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphz60pspa 2022-12-01T10:40:43.1472594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphz60pspa/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1473203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1473458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1474090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1474367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1474769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11h2yc7o 2022-12-01T10:40:43.1475190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11h2yc7o/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1475800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1476109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1476737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1477032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1477436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnm11c9s1 2022-12-01T10:40:43.1477892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnm11c9s1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1478327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1478680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1479037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1479390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1479532Z ok (2.140s) 2022-12-01T10:40:43.1479542Z 2022-12-01T10:40:43.1479981Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1480145Z Ran 1 test in 2.140s 2022-12-01T10:40:43.1480157Z 2022-12-01T10:40:43.1480285Z OK 2022-12-01T10:40:43.1480294Z 2022-12-01T10:40:43.1480454Z Generating XML reports... 2022-12-01T10:40:43.1481420Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103619.xml 2022-12-01T10:40:43.1482031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1482290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1482917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1483221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1483630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbu7bswe0 2022-12-01T10:40:43.1484059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbu7bswe0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1484631Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1484646Z 2022-12-01T10:40:43.1484776Z Running tests... 2022-12-01T10:40:43.1485201Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1485850Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25733 2022-12-01T10:40:43.1486191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25734 2022-12-01T10:40:43.1486534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25735 2022-12-01T10:40:43.1486869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25736 2022-12-01T10:40:43.1487478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1487785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1488428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1488708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1489113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9ws8fem 2022-12-01T10:40:43.1489544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9ws8fem/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1490132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1490463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1491084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1491368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1491768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vmh0rdr 2022-12-01T10:40:43.1492398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1492653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1493259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1493547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1493948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsj9y0h9 2022-12-01T10:40:43.1494220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vmh0rdr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1494412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsj9y0h9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1494684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1494810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1495077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1495212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1495395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp449yxlw3 2022-12-01T10:40:43.1495590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp449yxlw3/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1495754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1495918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1496080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1496241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1496297Z ok (2.338s) 2022-12-01T10:40:43.1496316Z 2022-12-01T10:40:43.1496533Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1496608Z Ran 1 test in 2.339s 2022-12-01T10:40:43.1496613Z 2022-12-01T10:40:43.1496673Z OK 2022-12-01T10:40:43.1496678Z 2022-12-01T10:40:43.1496762Z Generating XML reports... 2022-12-01T10:40:43.1497187Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103623.xml 2022-12-01T10:40:43.1497459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1497582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1497860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1497982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1498168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l3ll49e 2022-12-01T10:40:43.1498362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l3ll49e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1498618Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1498624Z 2022-12-01T10:40:43.1498697Z Running tests... 2022-12-01T10:40:43.1498933Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1499221Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25988 2022-12-01T10:40:43.1499379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25989 2022-12-01T10:40:43.1499534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25990 2022-12-01T10:40:43.1499735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25991 2022-12-01T10:40:43.1500011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1500133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1500411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1500549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1500735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjf26pzs 2022-12-01T10:40:43.1500934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjf26pzs/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1501209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1501320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1501605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1501739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1501924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw74mt1o9 2022-12-01T10:40:43.1502118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw74mt1o9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1502391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1502512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1502791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1502925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1503099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy05g17f4 2022-12-01T10:40:43.1503290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy05g17f4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1503559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1503682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1503964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1504098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1504280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3c1u02vi 2022-12-01T10:40:43.1504473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3c1u02vi/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1504626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1504791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1504952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1505113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1505182Z ok (2.138s) 2022-12-01T10:40:43.1505187Z 2022-12-01T10:40:43.1505414Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1505490Z Ran 1 test in 2.139s 2022-12-01T10:40:43.1505495Z 2022-12-01T10:40:43.1505556Z OK 2022-12-01T10:40:43.1505561Z 2022-12-01T10:40:43.1505644Z Generating XML reports... 2022-12-01T10:40:43.1506059Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103626.xml 2022-12-01T10:40:43.1506361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1506487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1506768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1506905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1507091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpql166rno 2022-12-01T10:40:43.1507289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpql166rno/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1507547Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1507552Z 2022-12-01T10:40:43.1507625Z Running tests... 2022-12-01T10:40:43.1507807Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1508096Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26243 2022-12-01T10:40:43.1508253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26244 2022-12-01T10:40:43.1508407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26245 2022-12-01T10:40:43.1508561Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26246 2022-12-01T10:40:43.1508839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1508963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1509245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1509368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1509556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv2dgyzx 2022-12-01T10:40:43.1509753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv2dgyzx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1510024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1510147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1510425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1510560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1510744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuerydtt 2022-12-01T10:40:43.1510940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuerydtt/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1511196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1511321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1511595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1511730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1511955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnxp06ru 2022-12-01T10:40:43.1512150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnxp06ru/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1512420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1512543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1512807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1512970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1513154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaaycrext 2022-12-01T10:40:43.1513348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaaycrext/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1513512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1513678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1513841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1514002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1514070Z ok (2.238s) 2022-12-01T10:40:43.1514075Z 2022-12-01T10:40:43.1514262Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1514341Z Ran 1 test in 2.239s 2022-12-01T10:40:43.1514346Z 2022-12-01T10:40:43.1514406Z OK 2022-12-01T10:40:43.1514411Z 2022-12-01T10:40:43.1514496Z Generating XML reports... 2022-12-01T10:40:43.1514924Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103630.xml 2022-12-01T10:40:43.1515199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1515323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1515604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1515727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1515911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljdocwsv 2022-12-01T10:40:43.1516109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljdocwsv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1516364Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1516369Z 2022-12-01T10:40:43.1516445Z Running tests... 2022-12-01T10:40:43.1516640Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1516937Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26498 2022-12-01T10:40:43.1517095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26499 2022-12-01T10:40:43.1517249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26500 2022-12-01T10:40:43.1517388Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26501 2022-12-01T10:40:43.1517662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1517789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1518069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1518308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1518544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8d6rmud 2022-12-01T10:40:43.1518744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8d6rmud/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1519020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1519143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1519411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1519588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1519771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37_b_u91 2022-12-01T10:40:43.1519964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37_b_u91/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1520236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1520364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1520643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1520777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1520948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprtzln6h 2022-12-01T10:40:43.1521145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprtzln6h/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1521414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1521536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1521816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1521952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1522138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpwfbrfe 2022-12-01T10:40:43.1522335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpwfbrfe/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1522497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1522649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1522813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1522973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1523043Z ok (2.139s) 2022-12-01T10:40:43.1523049Z 2022-12-01T10:40:43.1523247Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1523322Z Ran 1 test in 2.140s 2022-12-01T10:40:43.1523329Z 2022-12-01T10:40:43.1523389Z OK 2022-12-01T10:40:43.1523393Z 2022-12-01T10:40:43.1523476Z Generating XML reports... 2022-12-01T10:40:43.1523889Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103634.xml 2022-12-01T10:40:43.1524160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1524287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1524566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1524701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1524885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwil7iuwl 2022-12-01T10:40:43.1525107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwil7iuwl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1525366Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1525371Z 2022-12-01T10:40:43.1525444Z Running tests... 2022-12-01T10:40:43.1525628Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1525921Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26753 2022-12-01T10:40:43.1526107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26754 2022-12-01T10:40:43.1526262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26755 2022-12-01T10:40:43.1526418Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26756 2022-12-01T10:40:43.1526690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1526816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1527096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1527230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1527402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09a5ng00 2022-12-01T10:40:43.1527597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09a5ng00/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1527931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1528055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1528339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1528477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1528663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt423enoc 2022-12-01T10:40:43.1528857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt423enoc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1529112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1529240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1529520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1529655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1529838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2_0gyif 2022-12-01T10:40:43.1530031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2_0gyif/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1530302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1530425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1530703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1530824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1531008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3c2o5l2x 2022-12-01T10:40:43.1531199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3c2o5l2x/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1531363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1531526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1531721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1531884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1532062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1532226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1532555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1532728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1532898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1533194Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1533491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1533778Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1533846Z ok (2.238s) 2022-12-01T10:40:43.1533851Z 2022-12-01T10:40:43.1534047Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1534112Z Ran 1 test in 2.239s 2022-12-01T10:40:43.1534129Z 2022-12-01T10:40:43.1534176Z OK 2022-12-01T10:40:43.1534181Z 2022-12-01T10:40:43.1534265Z Generating XML reports... 2022-12-01T10:40:43.1534690Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103637.xml 2022-12-01T10:40:43.1534964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1535093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1535372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1535508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1535692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphstqbdr9 2022-12-01T10:40:43.1535879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphstqbdr9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1536135Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1536140Z 2022-12-01T10:40:43.1536212Z Running tests... 2022-12-01T10:40:43.1536409Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1536706Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27000 2022-12-01T10:40:43.1536863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27001 2022-12-01T10:40:43.1537019Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27002 2022-12-01T10:40:43.1537174Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27003 2022-12-01T10:40:43.1537448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1537561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1537842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1537978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1538161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6_5x7x2 2022-12-01T10:40:43.1538385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6_5x7x2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1538660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1538782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1539064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1539212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1539396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64if6y5e 2022-12-01T10:40:43.1539590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64if6y5e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1539859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1539984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1540261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1540396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1540580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokt7nxdm 2022-12-01T10:40:43.1540775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokt7nxdm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1541034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1541157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1541434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1541572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1541754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpch1vvc8f 2022-12-01T10:40:43.1541948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpch1vvc8f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1542111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1542275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1542427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1542588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1542766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1542941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1543118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1543417Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1543590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1543890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1544187Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1544478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1544533Z ok (2.242s) 2022-12-01T10:40:43.1544538Z 2022-12-01T10:40:43.1544737Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1544851Z Ran 1 test in 2.242s 2022-12-01T10:40:43.1544856Z 2022-12-01T10:40:43.1544917Z OK 2022-12-01T10:40:43.1544922Z 2022-12-01T10:40:43.1545007Z Generating XML reports... 2022-12-01T10:40:43.1545429Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103641.xml 2022-12-01T10:40:43.1545705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1545858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1546125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1546262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1546448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpig8vb3ld 2022-12-01T10:40:43.1546646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpig8vb3ld/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1546902Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1546908Z 2022-12-01T10:40:43.1546981Z Running tests... 2022-12-01T10:40:43.1547177Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1547479Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27247 2022-12-01T10:40:43.1547640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27248 2022-12-01T10:40:43.1547782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27249 2022-12-01T10:40:43.1547935Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27250 2022-12-01T10:40:43.1548211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1548336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1548614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1548752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1548941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpobae9p 2022-12-01T10:40:43.1549139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpobae9p/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1549409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1549519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1549800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1549935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1550119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6r4yd57g 2022-12-01T10:40:43.1550314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6r4yd57g/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1550585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1550711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1550989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1551111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1551294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulgky9qe 2022-12-01T10:40:43.1551532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulgky9qe/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1551805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1551928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1552205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1552368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1552551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20pijcxb 2022-12-01T10:40:43.1552745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20pijcxb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1552895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1553062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1553226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1553390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1553567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1553743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1553920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1554223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1554381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1554683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1554977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1555315Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1555384Z ok (2.240s) 2022-12-01T10:40:43.1555389Z 2022-12-01T10:40:43.1555590Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1555666Z Ran 1 test in 2.240s 2022-12-01T10:40:43.1555671Z 2022-12-01T10:40:43.1555730Z OK 2022-12-01T10:40:43.1555735Z 2022-12-01T10:40:43.1555819Z Generating XML reports... 2022-12-01T10:40:43.1556228Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103645.xml 2022-12-01T10:40:43.1556505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1556630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1556911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1557048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1557237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsh3bx1l 2022-12-01T10:40:43.1557436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsh3bx1l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1557691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1557697Z 2022-12-01T10:40:43.1557769Z Running tests... 2022-12-01T10:40:43.1557952Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1558421Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27494 2022-12-01T10:40:43.1558583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27495 2022-12-01T10:40:43.1558740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27496 2022-12-01T10:40:43.1558895Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27497 2022-12-01T10:40:43.1559211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1559340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1559622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1559757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1559933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel795pfc 2022-12-01T10:40:43.1560131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel795pfc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1560407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1560530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1560812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1560949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1561134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzdawljt 2022-12-01T10:40:43.1561329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzdawljt/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1561587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1561712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1561990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1562124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1562306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrmo_h6c 2022-12-01T10:40:43.1562502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrmo_h6c/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1562772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1562894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1563173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1563294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1563479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzes7a3pb 2022-12-01T10:40:43.1563671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzes7a3pb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1563837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1564004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1564168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1564329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1564396Z ok (2.140s) 2022-12-01T10:40:43.1564401Z 2022-12-01T10:40:43.1564586Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1564695Z Ran 1 test in 2.140s 2022-12-01T10:40:43.1564701Z 2022-12-01T10:40:43.1564763Z OK 2022-12-01T10:40:43.1564767Z 2022-12-01T10:40:43.1564850Z Generating XML reports... 2022-12-01T10:40:43.1565278Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103648.xml 2022-12-01T10:40:43.1565548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1565700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1565978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1566113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1566284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jtfj86l 2022-12-01T10:40:43.1566479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jtfj86l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1566731Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1566736Z 2022-12-01T10:40:43.1566809Z Running tests... 2022-12-01T10:40:43.1567003Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1567290Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27761 2022-12-01T10:40:43.1567450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27762 2022-12-01T10:40:43.1567607Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27763 2022-12-01T10:40:43.1567848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27764 2022-12-01T10:40:43.1568122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1568247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1568532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1568668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1568854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahe8e5ye 2022-12-01T10:40:43.1569053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahe8e5ye/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1569326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1569451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1569718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1569857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1570043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2lzb79n 2022-12-01T10:40:43.1570240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2lzb79n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1570516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1570643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1570922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1571059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1571244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwymk5qqm 2022-12-01T10:40:43.1571478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwymk5qqm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1571749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1571871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1572151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1572310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1572494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2qzv9gx 2022-12-01T10:40:43.1572688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2qzv9gx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1572851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1573005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1573167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1573329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1573505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1573683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1573859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1574033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1574333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1574633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1574909Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1575197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1575265Z ok (2.439s) 2022-12-01T10:40:43.1575270Z 2022-12-01T10:40:43.1575466Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1575545Z Ran 1 test in 2.439s 2022-12-01T10:40:43.1575550Z 2022-12-01T10:40:43.1575610Z OK 2022-12-01T10:40:43.1575614Z 2022-12-01T10:40:43.1575697Z Generating XML reports... 2022-12-01T10:40:43.1576124Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103652.xml 2022-12-01T10:40:43.1576399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1576510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1576790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1576925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1577111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflnhgfzi 2022-12-01T10:40:43.1577311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflnhgfzi/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1577563Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1577568Z 2022-12-01T10:40:43.1577641Z Running tests... 2022-12-01T10:40:43.1577835Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1578158Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28008 2022-12-01T10:40:43.1578305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28009 2022-12-01T10:40:43.1578462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28010 2022-12-01T10:40:43.1578616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28011 2022-12-01T10:40:43.1578920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1579043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1579325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1579461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1579650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nn6ipg9 2022-12-01T10:40:43.1579845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nn6ipg9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1580102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1580223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1580500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1580636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1580819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q3y1y8m 2022-12-01T10:40:43.1581086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1581208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1581489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1581612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1581795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbmoorw1 2022-12-01T10:40:43.1581987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q3y1y8m/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1582180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbmoorw1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1582454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1582577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1582857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1582993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1583177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3ih1zrn 2022-12-01T10:40:43.1583359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3ih1zrn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1583521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1583688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1583850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1584012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1584188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1584393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1584568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1584730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1585030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1585356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1585648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1585938Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1586006Z ok (2.440s) 2022-12-01T10:40:43.1586011Z 2022-12-01T10:40:43.1586211Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1586287Z Ran 1 test in 2.440s 2022-12-01T10:40:43.1586293Z 2022-12-01T10:40:43.1586353Z OK 2022-12-01T10:40:43.1586359Z 2022-12-01T10:40:43.1586430Z Generating XML reports... 2022-12-01T10:40:43.1586858Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103656.xml 2022-12-01T10:40:43.1587132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1587259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1587540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1587675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1587867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp3o1ygp 2022-12-01T10:40:43.1588067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp3o1ygp/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1588320Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1588325Z 2022-12-01T10:40:43.1588387Z Running tests... 2022-12-01T10:40:43.1588581Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1588877Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28255 2022-12-01T10:40:43.1589037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28256 2022-12-01T10:40:43.1589192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28257 2022-12-01T10:40:43.1589347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28258 2022-12-01T10:40:43.1589627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1589750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1590030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1590153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1590344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlxzrp2_ 2022-12-01T10:40:43.1590539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlxzrp2_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1590811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1590933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1591251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1591385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1591570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxj31pjjx 2022-12-01T10:40:43.1591750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxj31pjjx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1592020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1592171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1592448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1592584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1592768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjuxar9q 2022-12-01T10:40:43.1593036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1593159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1593434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1593555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1593741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyapxrno 2022-12-01T10:40:43.1593934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjuxar9q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1594125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyapxrno/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1594290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1594456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1594617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1594778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1594941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1595121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1595295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1595468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1595766Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1596064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1596355Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1596643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1596711Z ok (2.342s) 2022-12-01T10:40:43.1596718Z 2022-12-01T10:40:43.1596914Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1596977Z Ran 1 test in 2.343s 2022-12-01T10:40:43.1596982Z 2022-12-01T10:40:43.1597045Z OK 2022-12-01T10:40:43.1597050Z 2022-12-01T10:40:43.1597134Z Generating XML reports... 2022-12-01T10:40:43.1597563Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103700.xml 2022-12-01T10:40:43.1597863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1597987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1598398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1598535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1598756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0obwyw7k 2022-12-01T10:40:43.1598951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0obwyw7k/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1599206Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1599211Z 2022-12-01T10:40:43.1599286Z Running tests... 2022-12-01T10:40:43.1599483Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1599775Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28502 2022-12-01T10:40:43.1599935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28503 2022-12-01T10:40:43.1600091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28504 2022-12-01T10:40:43.1600248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28505 2022-12-01T10:40:43.1600508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1600634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1600914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1601049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1601239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphoohppuz 2022-12-01T10:40:43.1601436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphoohppuz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1601707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1601831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1602110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1602231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1602415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvnqzz2e 2022-12-01T10:40:43.1602609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvnqzz2e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1602882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1603004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1603280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1603414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1603596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsp6dj1ya 2022-12-01T10:40:43.1603780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsp6dj1ya/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1604048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1604171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1604483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1604619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1604802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna9m3qe6 2022-12-01T10:40:43.1604996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna9m3qe6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1605159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1605351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1605501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1605661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1605837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1606017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1606318Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1606488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1606660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1606955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1607249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1607527Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1607596Z ok (2.253s) 2022-12-01T10:40:43.1607603Z 2022-12-01T10:40:43.1607856Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1607934Z Ran 1 test in 2.253s 2022-12-01T10:40:43.1607939Z 2022-12-01T10:40:43.1608000Z OK 2022-12-01T10:40:43.1608005Z 2022-12-01T10:40:43.1608092Z Generating XML reports... 2022-12-01T10:40:43.1608524Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103703.xml 2022-12-01T10:40:43.1608802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1608914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1609194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1609330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1609519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvia0_gw 2022-12-01T10:40:43.1609716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvia0_gw/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1609970Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1609975Z 2022-12-01T10:40:43.1610050Z Running tests... 2022-12-01T10:40:43.1610248Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1610554Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28749 2022-12-01T10:40:43.1610699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28750 2022-12-01T10:40:43.1610855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28751 2022-12-01T10:40:43.1611046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28752 2022-12-01T10:40:43.1611325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1611448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1611730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1611893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1612082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkug2s1gp 2022-12-01T10:40:43.1612280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkug2s1gp/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1612541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1612664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1612943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1613080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1613265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn3y32qo 2022-12-01T10:40:43.1613460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn3y32qo/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1613732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1613854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1614121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1614255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1614438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_s_gz98 2022-12-01T10:40:43.1614630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_s_gz98/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1614901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1615025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1615306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1615440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1615620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kqn6ewa 2022-12-01T10:40:43.1615799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kqn6ewa/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1615964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1616128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1616291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1616451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1616629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1616808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1616981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1617280Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1617465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1617761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1618057Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1618348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1618443Z ok (2.448s) 2022-12-01T10:40:43.1618448Z 2022-12-01T10:40:43.1618645Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1618723Z Ran 1 test in 2.448s 2022-12-01T10:40:43.1618728Z 2022-12-01T10:40:43.1618787Z OK 2022-12-01T10:40:43.1618792Z 2022-12-01T10:40:43.1618876Z Generating XML reports... 2022-12-01T10:40:43.1619291Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103707.xml 2022-12-01T10:40:43.1619566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1619688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1619969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1620106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1620294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99q07v10 2022-12-01T10:40:43.1620489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99q07v10/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1620743Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1620748Z 2022-12-01T10:40:43.1620808Z Running tests... 2022-12-01T10:40:43.1621008Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1621306Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28996 2022-12-01T10:40:43.1621465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28997 2022-12-01T10:40:43.1621625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28998 2022-12-01T10:40:43.1621781Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28999 2022-12-01T10:40:43.1622054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1622178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1622458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1622584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1622769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp484d0ao2 2022-12-01T10:40:43.1622963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp484d0ao2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1623233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1623358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1623637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1623771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1623952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_q1275n8 2022-12-01T10:40:43.1624168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_q1275n8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1624429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1624551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1624833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1624967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1625179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4exysa9 2022-12-01T10:40:43.1625373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4exysa9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1625645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1625767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1626035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1626168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1626352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpayfhcjnf 2022-12-01T10:40:43.1626545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpayfhcjnf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1626710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1626872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1627034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1627195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1627374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1627538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1627711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1628012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1628186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1628483Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1628780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1629072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1629140Z ok (2.343s) 2022-12-01T10:40:43.1629144Z 2022-12-01T10:40:43.1629343Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1629407Z Ran 1 test in 2.343s 2022-12-01T10:40:43.1629412Z 2022-12-01T10:40:43.1629473Z OK 2022-12-01T10:40:43.1629478Z 2022-12-01T10:40:43.1629561Z Generating XML reports... 2022-12-01T10:40:43.1629991Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103711.xml 2022-12-01T10:40:43.1630268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1630392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1630672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1630834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1631009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8whqwq63 2022-12-01T10:40:43.1631203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8whqwq63/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1631458Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1631493Z 2022-12-01T10:40:43.1631567Z Running tests... 2022-12-01T10:40:43.1631766Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1632053Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29243 2022-12-01T10:40:43.1632210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29244 2022-12-01T10:40:43.1632367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29245 2022-12-01T10:40:43.1632521Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29246 2022-12-01T10:40:43.1632781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1632905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1633185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1633325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1633509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr36tluva 2022-12-01T10:40:43.1633705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr36tluva/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1633977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1634104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1634386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1634507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1634692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvw558po 2022-12-01T10:40:43.1634963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1635084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1635362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1635494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1635684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7jrbf__ 2022-12-01T10:40:43.1635880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvw558po/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1636058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7jrbf__/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1636329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1636453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1636729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1636863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1637045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24qldecn 2022-12-01T10:40:43.1637278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24qldecn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1637442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1637603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1637752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1637912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1638116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1638397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1638574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1638874Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1639047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1639348Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1639644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1639923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1639996Z ok (2.448s) 2022-12-01T10:40:43.1640001Z 2022-12-01T10:40:43.1640200Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1640275Z Ran 1 test in 2.448s 2022-12-01T10:40:43.1640281Z 2022-12-01T10:40:43.1640343Z OK 2022-12-01T10:40:43.1640348Z 2022-12-01T10:40:43.1640432Z Generating XML reports... 2022-12-01T10:40:43.1640863Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103715.xml 2022-12-01T10:40:43.1641136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1641261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1641530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1641670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1641855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wnslhk1 2022-12-01T10:40:43.1642052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wnslhk1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1642309Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1642314Z 2022-12-01T10:40:43.1642390Z Running tests... 2022-12-01T10:40:43.1642586Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1642881Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29490 2022-12-01T10:40:43.1643044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29491 2022-12-01T10:40:43.1643191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29492 2022-12-01T10:40:43.1643345Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29493 2022-12-01T10:40:43.1643621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1643745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1644076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1644214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1644400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvw91fs2 2022-12-01T10:40:43.1644596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvw91fs2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1644856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1645015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1645297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1645434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1645619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppndg7n59 2022-12-01T10:40:43.1645816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppndg7n59/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1646087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1646210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1646489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1646614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1646798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32z5q1v7 2022-12-01T10:40:43.1646991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32z5q1v7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1647263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1647387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1647666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1647856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1648041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_2xgyzi 2022-12-01T10:40:43.1648221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_2xgyzi/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1648393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1648554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1648717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1648882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1649062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1649239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1649414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1649716Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1649879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1650174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1650473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1650800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1650869Z ok (2.453s) 2022-12-01T10:40:43.1650875Z 2022-12-01T10:40:43.1651071Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1651146Z Ran 1 test in 2.454s 2022-12-01T10:40:43.1651152Z 2022-12-01T10:40:43.1651212Z OK 2022-12-01T10:40:43.1651216Z 2022-12-01T10:40:43.1651301Z Generating XML reports... 2022-12-01T10:40:43.1651763Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103719.xml 2022-12-01T10:40:43.1652034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1652158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1652442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1652580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1652765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvv6xahf 2022-12-01T10:40:43.1652961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvv6xahf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1653215Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1653223Z 2022-12-01T10:40:43.1653295Z Running tests... 2022-12-01T10:40:43.1653477Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1653765Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29737 2022-12-01T10:40:43.1653921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29738 2022-12-01T10:40:43.1654082Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29739 2022-12-01T10:40:43.1654235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29740 2022-12-01T10:40:43.1654511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1654635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1654915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1655040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1655222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8qk4l3j 2022-12-01T10:40:43.1655416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8qk4l3j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1655691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1655815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1656095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1656229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1656415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbnkjoa5 2022-12-01T10:40:43.1656615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbnkjoa5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1656872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1656994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1657303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1657441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1657624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmau_o6x3 2022-12-01T10:40:43.1657816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmau_o6x3/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1658085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1658233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1658499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1658634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1658816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69gyga1v 2022-12-01T10:40:43.1659010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69gyga1v/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1659174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1659336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1659497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1659658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1659729Z ok (2.244s) 2022-12-01T10:40:43.1659735Z 2022-12-01T10:40:43.1659921Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1659996Z Ran 1 test in 2.244s 2022-12-01T10:40:43.1660001Z 2022-12-01T10:40:43.1660060Z OK 2022-12-01T10:40:43.1660065Z 2022-12-01T10:40:43.1660148Z Generating XML reports... 2022-12-01T10:40:43.1660579Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103723.xml 2022-12-01T10:40:43.1660850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1660976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1661255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1661379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1661563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawj22g99 2022-12-01T10:40:43.1661755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawj22g99/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1662009Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1662014Z 2022-12-01T10:40:43.1662085Z Running tests... 2022-12-01T10:40:43.1662284Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1662568Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29992 2022-12-01T10:40:43.1662727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29993 2022-12-01T10:40:43.1662881Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29994 2022-12-01T10:40:43.1663024Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29995 2022-12-01T10:40:43.1663303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1663428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1663709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1663870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1664057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoyf6l6ed 2022-12-01T10:40:43.1664253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoyf6l6ed/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1664525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1664674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1664942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1665077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1665261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj5g_5ry 2022-12-01T10:40:43.1665455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj5g_5ry/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1665723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1665846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1666123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1666260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1666431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5d0jbyl 2022-12-01T10:40:43.1666625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5d0jbyl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1666893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1667017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1667294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1667427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1667610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ah8oth7 2022-12-01T10:40:43.1667802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ah8oth7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1667967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1668119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1668286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1668448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1668516Z ok (3.249s) 2022-12-01T10:40:43.1668523Z 2022-12-01T10:40:43.1668719Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1668794Z Ran 1 test in 3.249s 2022-12-01T10:40:43.1668799Z 2022-12-01T10:40:43.1668859Z OK 2022-12-01T10:40:43.1668864Z 2022-12-01T10:40:43.1668947Z Generating XML reports... 2022-12-01T10:40:43.1669360Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103726.xml 2022-12-01T10:40:43.1669635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1669759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1670038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1670172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1670398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlqpd9zm 2022-12-01T10:40:43.1670600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlqpd9zm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1670859Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1670864Z 2022-12-01T10:40:43.1670936Z Running tests... 2022-12-01T10:40:43.1671120Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1671447Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30247 2022-12-01T10:40:43.1671606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30248 2022-12-01T10:40:43.1671765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30249 2022-12-01T10:40:43.1671923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30250 2022-12-01T10:40:43.1672196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1672319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1672597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1672733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1672907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzusoy42_ 2022-12-01T10:40:43.1673101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzusoy42_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1673370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1673494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1673775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1673909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1674093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmab6242 2022-12-01T10:40:43.1674287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmab6242/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1674545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1674667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1674944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1675079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1675262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8x_cl1g 2022-12-01T10:40:43.1675454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8x_cl1g/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1675724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1675848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1676125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1676246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1676430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2eynsdfj 2022-12-01T10:40:43.1676627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2eynsdfj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1676821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1676985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1677149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1677310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1677379Z ok (2.248s) 2022-12-01T10:40:43.1677410Z 2022-12-01T10:40:43.1677597Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1677673Z Ran 1 test in 2.249s 2022-12-01T10:40:43.1677678Z 2022-12-01T10:40:43.1677737Z OK 2022-12-01T10:40:43.1677742Z 2022-12-01T10:40:43.1677826Z Generating XML reports... 2022-12-01T10:40:43.1678369Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103731.xml 2022-12-01T10:40:43.1678647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1678772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1679054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1679191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1679365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueeblxas 2022-12-01T10:40:43.1679562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueeblxas/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1679819Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1679824Z 2022-12-01T10:40:43.1679897Z Running tests... 2022-12-01T10:40:43.1680095Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1680405Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30502 2022-12-01T10:40:43.1680565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30503 2022-12-01T10:40:43.1680721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30504 2022-12-01T10:40:43.1680876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30505 2022-12-01T10:40:43.1681139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1681264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1681545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1681681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1681870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_jyhv_p 2022-12-01T10:40:43.1682068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_jyhv_p/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1682340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1682463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1682729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1682863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1683047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxc60gfm 2022-12-01T10:40:43.1683241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxc60gfm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1683577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1683706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1683987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1684122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1684305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn04ux342 2022-12-01T10:40:43.1684595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1684720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1684999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1685135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1685319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7r9ao2k7 2022-12-01T10:40:43.1685512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn04ux342/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1685701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7r9ao2k7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1685866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1686020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1686182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1686344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1686523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1686703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1686882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1687056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1687357Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1687660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1687997Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1688292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1688362Z ok (2.349s) 2022-12-01T10:40:43.1688369Z 2022-12-01T10:40:43.1688567Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1688643Z Ran 1 test in 2.349s 2022-12-01T10:40:43.1688648Z 2022-12-01T10:40:43.1688710Z OK 2022-12-01T10:40:43.1688714Z 2022-12-01T10:40:43.1688800Z Generating XML reports... 2022-12-01T10:40:43.1689229Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103735.xml 2022-12-01T10:40:43.1689509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1689621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1689905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1690041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1690260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_t8z05lc 2022-12-01T10:40:43.1690455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_t8z05lc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1690712Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1690718Z 2022-12-01T10:40:43.1690791Z Running tests... 2022-12-01T10:40:43.1690990Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1691318Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30749 2022-12-01T10:40:43.1691463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30750 2022-12-01T10:40:43.1691616Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30751 2022-12-01T10:40:43.1691772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30752 2022-12-01T10:40:43.1692047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1692172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1692452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1692589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1692776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoc6zwa_i 2022-12-01T10:40:43.1692973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoc6zwa_i/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1693229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1693352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1693634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1693770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1693955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1eyw0yv1 2022-12-01T10:40:43.1694151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1eyw0yv1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1694424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1694548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1694812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1694945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1695131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumhlro6o 2022-12-01T10:40:43.1695329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumhlro6o/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1695597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1695719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1695998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1696130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1696315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuae3h6sv 2022-12-01T10:40:43.1696495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuae3h6sv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1696687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1696852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1697014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1697174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1697350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1697561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1697735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1697891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1698194Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1698492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1698785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1699079Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1699154Z ok (2.350s) 2022-12-01T10:40:43.1699159Z 2022-12-01T10:40:43.1699356Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1699432Z Ran 1 test in 2.351s 2022-12-01T10:40:43.1699437Z 2022-12-01T10:40:43.1699496Z OK 2022-12-01T10:40:43.1699501Z 2022-12-01T10:40:43.1699573Z Generating XML reports... 2022-12-01T10:40:43.1700006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103739.xml 2022-12-01T10:40:43.1700280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1700405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1700689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1700826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1701019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy17642ij 2022-12-01T10:40:43.1701213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy17642ij/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1701466Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1701471Z 2022-12-01T10:40:43.1701531Z Running tests... 2022-12-01T10:40:43.1701727Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1702013Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30996 2022-12-01T10:40:43.1702170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30997 2022-12-01T10:40:43.1702326Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30998 2022-12-01T10:40:43.1702481Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30999 2022-12-01T10:40:43.1702757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1702880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1703161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1703311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1703498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphuatgnu8 2022-12-01T10:40:43.1703693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphuatgnu8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1703963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1704115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1704395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1704529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1704716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6bu5jc2 2022-12-01T10:40:43.1704898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6bu5jc2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1705168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1705292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1705568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1705702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1705889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sw8wah2 2022-12-01T10:40:43.1706082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sw8wah2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1706351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1706473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1706739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1706872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1707053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp948l1_g9 2022-12-01T10:40:43.1707242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp948l1_g9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1707407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1707571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1707732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1707893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1707949Z ok (2.246s) 2022-12-01T10:40:43.1707967Z 2022-12-01T10:40:43.1708155Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1708231Z Ran 1 test in 2.246s 2022-12-01T10:40:43.1708236Z 2022-12-01T10:40:43.1708299Z OK 2022-12-01T10:40:43.1708304Z 2022-12-01T10:40:43.1708389Z Generating XML reports... 2022-12-01T10:40:43.1708814Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103743.xml 2022-12-01T10:40:43.1709085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1709208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1709486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1709607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1709817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwjzimov 2022-12-01T10:40:43.1710012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwjzimov/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1710266Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1710272Z 2022-12-01T10:40:43.1710345Z Running tests... 2022-12-01T10:40:43.1710541Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1710856Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31243 2022-12-01T10:40:43.1711013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31244 2022-12-01T10:40:43.1711168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31245 2022-12-01T10:40:43.1711309Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31246 2022-12-01T10:40:43.1711588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1711711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1712003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1712138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1712329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkfdh0nj 2022-12-01T10:40:43.1712527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkfdh0nj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1712787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1712912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1713195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1713329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1713512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1n9nl3w5 2022-12-01T10:40:43.1713706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1n9nl3w5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1713980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1714103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1714385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1714507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1714692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1y36n5j8 2022-12-01T10:40:43.1714882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1y36n5j8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1715153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1715276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1715554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1715690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1715875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0rzm4l0 2022-12-01T10:40:43.1716057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0rzm4l0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1716219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1716407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1716573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1716735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1716806Z ok (2.255s) 2022-12-01T10:40:43.1716811Z 2022-12-01T10:40:43.1717013Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1717116Z Ran 1 test in 2.255s 2022-12-01T10:40:43.1717121Z 2022-12-01T10:40:43.1717169Z OK 2022-12-01T10:40:43.1717186Z 2022-12-01T10:40:43.1717256Z Generating XML reports... 2022-12-01T10:40:43.1717685Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103746.xml 2022-12-01T10:40:43.1717959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1718085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1718487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1718624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1718811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6khdsbv 2022-12-01T10:40:43.1719013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6khdsbv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1719257Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1719277Z 2022-12-01T10:40:43.1719337Z Running tests... 2022-12-01T10:40:43.1719533Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1719812Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31490 2022-12-01T10:40:43.1719972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31491 2022-12-01T10:40:43.1720129Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31492 2022-12-01T10:40:43.1720284Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31493 2022-12-01T10:40:43.1720563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1720691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1720960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1721096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1721285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69z8yf8o 2022-12-01T10:40:43.1721480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69z8yf8o/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1721754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1721878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1722161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1722299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1722471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot0pyx5g 2022-12-01T10:40:43.1722668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot0pyx5g/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1722985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1723110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1723389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1723525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1723710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxrj2bzj 2022-12-01T10:40:43.1723938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxrj2bzj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1724210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1724319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1724598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1724735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1724917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4g_2wf_k 2022-12-01T10:40:43.1725110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4g_2wf_k/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1725276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1725444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1725610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1725774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1725829Z ok (2.347s) 2022-12-01T10:40:43.1725834Z 2022-12-01T10:40:43.1726034Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1726114Z Ran 1 test in 2.348s 2022-12-01T10:40:43.1726122Z 2022-12-01T10:40:43.1726186Z OK 2022-12-01T10:40:43.1726191Z 2022-12-01T10:40:43.1726278Z Generating XML reports... 2022-12-01T10:40:43.1726706Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103750.xml 2022-12-01T10:40:43.1726977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1727102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1727367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1727502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1727738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzm5i70c6 2022-12-01T10:40:43.1727943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzm5i70c6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1728202Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1728207Z 2022-12-01T10:40:43.1728281Z Running tests... 2022-12-01T10:40:43.1728479Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1728744Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31757 2022-12-01T10:40:43.1728904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31758 2022-12-01T10:40:43.1729047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31759 2022-12-01T10:40:43.1729204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31760 2022-12-01T10:40:43.1729479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1729650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1729936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1730072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1730260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsollh8v 2022-12-01T10:40:43.1730497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsollh8v/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1730756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1730881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1731162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1731299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1731482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qj0_dqc 2022-12-01T10:40:43.1731678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qj0_dqc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1731946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1732069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1732347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1732466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1732648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl42ofv4j 2022-12-01T10:40:43.1732840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl42ofv4j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1733112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1733235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1733513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1733647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1733835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstbqspjj 2022-12-01T10:40:43.1734017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstbqspjj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1734181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1734343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1734506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1734666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1734733Z ok (2.150s) 2022-12-01T10:40:43.1734738Z 2022-12-01T10:40:43.1734934Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1735009Z Ran 1 test in 2.151s 2022-12-01T10:40:43.1735014Z 2022-12-01T10:40:43.1735075Z OK 2022-12-01T10:40:43.1735080Z 2022-12-01T10:40:43.1735152Z Generating XML reports... 2022-12-01T10:40:43.1735542Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103754.xml 2022-12-01T10:40:43.1735813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1735936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1736245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1736380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1736566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlm8ut8l 2022-12-01T10:40:43.1736761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlm8ut8l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1737036Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1737053Z 2022-12-01T10:40:43.1737113Z Running tests... 2022-12-01T10:40:43.1737310Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1737572Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31992 2022-12-01T10:40:43.1737731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31993 2022-12-01T10:40:43.1737888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31994 2022-12-01T10:40:43.1738040Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31995 2022-12-01T10:40:43.1738313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1738437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1738706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1738841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1739026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzch34lcz 2022-12-01T10:40:43.1739221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzch34lcz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1739491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1739613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1739893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1740027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1740214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24ptvx9z 2022-12-01T10:40:43.1740472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1740594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1740870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1741008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1741193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjc981s5k 2022-12-01T10:40:43.1741388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24ptvx9z/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1741582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjc981s5k/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1741855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1741969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1742249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1742382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1742565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mwp2hmu 2022-12-01T10:40:43.1742786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mwp2hmu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1742949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1743111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1743271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1743459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1743514Z ok (2.146s) 2022-12-01T10:40:43.1743519Z 2022-12-01T10:40:43.1743715Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1743790Z Ran 1 test in 2.146s 2022-12-01T10:40:43.1743795Z 2022-12-01T10:40:43.1743855Z OK 2022-12-01T10:40:43.1743860Z 2022-12-01T10:40:43.1743945Z Generating XML reports... 2022-12-01T10:40:43.1744339Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103758.xml 2022-12-01T10:40:43.1744609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1744734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1745002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1745139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1745323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwjv2n9f 2022-12-01T10:40:43.1745520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwjv2n9f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1745773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1745778Z 2022-12-01T10:40:43.1745854Z Running tests... 2022-12-01T10:40:43.1746051Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1746326Z test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32227 2022-12-01T10:40:43.1746482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32228 2022-12-01T10:40:43.1746627Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32229 2022-12-01T10:40:43.1746778Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32230 2022-12-01T10:40:43.1747049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1747173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1747453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1747588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1747773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoywr1rxt 2022-12-01T10:40:43.1747970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoywr1rxt/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1748242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1748355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1748631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1748765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1748949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p0n2far 2022-12-01T10:40:43.1749173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p0n2far/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1749444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1749567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1749846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1749992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1750175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qp3eaya 2022-12-01T10:40:43.1750366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qp3eaya/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1750633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1750758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1751036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1751170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1751354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfuielmb 2022-12-01T10:40:43.1751547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfuielmb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1751700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1751863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1752023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1752185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1752362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1752539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1752712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1753010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1753183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1753470Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1753766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1754059Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1754329Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1754596Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1754666Z ok (2.353s) 2022-12-01T10:40:43.1754672Z 2022-12-01T10:40:43.1754869Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1754945Z Ran 1 test in 2.353s 2022-12-01T10:40:43.1754950Z 2022-12-01T10:40:43.1755009Z OK 2022-12-01T10:40:43.1755014Z 2022-12-01T10:40:43.1755085Z Generating XML reports... 2022-12-01T10:40:43.1755528Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103801.xml 2022-12-01T10:40:43.1755807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1755933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1756213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1756375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1756561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5oiay9b 2022-12-01T10:40:43.1756757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5oiay9b/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1757011Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1757017Z 2022-12-01T10:40:43.1757076Z Running tests... 2022-12-01T10:40:43.1757272Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1758057Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84116 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.222s) 2022-12-01T10:40:43.1758065Z 2022-12-01T10:40:43.1758387Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1758464Z Ran 1 test in 0.222s 2022-12-01T10:40:43.1758470Z 2022-12-01T10:40:43.1758543Z OK (skipped=1) 2022-12-01T10:40:43.1758547Z 2022-12-01T10:40:43.1758633Z Generating XML reports... 2022-12-01T10:40:43.1759025Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103805.xml 2022-12-01T10:40:43.1759302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1759427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1759695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1759831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1760016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_vx574q 2022-12-01T10:40:43.1760211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_vx574q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1760464Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1760469Z 2022-12-01T10:40:43.1760543Z Running tests... 2022-12-01T10:40:43.1760742Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1761039Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32447 2022-12-01T10:40:43.1761197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32448 2022-12-01T10:40:43.1761340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32449 2022-12-01T10:40:43.1761495Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32450 2022-12-01T10:40:43.1761767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1761891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1762174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1762357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1762546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rimxkl_ 2022-12-01T10:40:43.1762741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rimxkl_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1763013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1763164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1763444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1763578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1763763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0et6sc6m 2022-12-01T10:40:43.1763960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0et6sc6m/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1764230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1764355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1764633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1764754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1764938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9074000p 2022-12-01T10:40:43.1765130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9074000p/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1765397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1765519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1765800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1765933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1766116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpzkbn5s 2022-12-01T10:40:43.1766308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpzkbn5s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1766462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1766625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1766788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1766951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1767057Z skip: Need at least 2 CUDA devices (1.859s) 2022-12-01T10:40:43.1767068Z 2022-12-01T10:40:43.1767263Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1767336Z Ran 1 test in 1.859s 2022-12-01T10:40:43.1767341Z 2022-12-01T10:40:43.1767412Z OK (skipped=1) 2022-12-01T10:40:43.1767416Z 2022-12-01T10:40:43.1767488Z Generating XML reports... 2022-12-01T10:40:43.1767941Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103807.xml 2022-12-01T10:40:43.1768227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1768352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1768631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1768768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1768987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cfttb63 2022-12-01T10:40:43.1769185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cfttb63/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1769444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1769450Z 2022-12-01T10:40:43.1769509Z Running tests... 2022-12-01T10:40:43.1769734Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1769992Z test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32502 2022-12-01T10:40:43.1770148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32503 2022-12-01T10:40:43.1770302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32504 2022-12-01T10:40:43.1770457Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32505 2022-12-01T10:40:43.1770732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1770855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1771136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1771258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1771447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmq1to4iv 2022-12-01T10:40:43.1771642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmq1to4iv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1771916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1772037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1772318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1772456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1772644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytzsc0c_ 2022-12-01T10:40:43.1772827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytzsc0c_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1773102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1773227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1773509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1773644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1773832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymu_8axp 2022-12-01T10:40:43.1774026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymu_8axp/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1774296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1774420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1774689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1774824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1775008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphs8e_u_r 2022-12-01T10:40:43.1775202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphs8e_u_r/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1775417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1775586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1775750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1775914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1775969Z ok (2.145s) 2022-12-01T10:40:43.1776003Z 2022-12-01T10:40:43.1776205Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1776283Z Ran 1 test in 2.146s 2022-12-01T10:40:43.1776288Z 2022-12-01T10:40:43.1776351Z OK 2022-12-01T10:40:43.1776356Z 2022-12-01T10:40:43.1776441Z Generating XML reports... 2022-12-01T10:40:43.1776835Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103810.xml 2022-12-01T10:40:43.1777114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1777239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1777522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1777643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1777830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr47bbxaf 2022-12-01T10:40:43.1778028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr47bbxaf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1778286Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1778291Z 2022-12-01T10:40:43.1778365Z Running tests... 2022-12-01T10:40:43.1778563Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1778835Z test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32713 2022-12-01T10:40:43.1778996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32714 2022-12-01T10:40:43.1779138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32715 2022-12-01T10:40:43.1779295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32716 2022-12-01T10:40:43.1779576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1779700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1779981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1780117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1780305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8a2kiz7 2022-12-01T10:40:43.1780501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8a2kiz7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1780775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1780887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1781167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1781305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1781485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jtc6llv 2022-12-01T10:40:43.1781678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jtc6llv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1781978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1782104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1782381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1782514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1782684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9nts61c 2022-12-01T10:40:43.1782906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9nts61c/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1783176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1783298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1783576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1783710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1783893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai50axne 2022-12-01T10:40:43.1784088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai50axne/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1784239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1784405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1784565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1784728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1784795Z ok (1.947s) 2022-12-01T10:40:43.1784801Z 2022-12-01T10:40:43.1784997Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1785073Z Ran 1 test in 1.947s 2022-12-01T10:40:43.1785078Z 2022-12-01T10:40:43.1785138Z OK 2022-12-01T10:40:43.1785143Z 2022-12-01T10:40:43.1785213Z Generating XML reports... 2022-12-01T10:40:43.1785602Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103814.xml 2022-12-01T10:40:43.1785877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1786003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1786285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1786420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1786605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfth0jzv 2022-12-01T10:40:43.1786804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfth0jzv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1787059Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1787064Z 2022-12-01T10:40:43.1787124Z Running tests... 2022-12-01T10:40:43.1787319Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1787609Z test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 304 2022-12-01T10:40:43.1787767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 305 2022-12-01T10:40:43.1787919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 306 2022-12-01T10:40:43.1788072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 307 2022-12-01T10:40:43.1788374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1788500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1788780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1788902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1789085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yqgnoqz 2022-12-01T10:40:43.1789307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yqgnoqz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1789578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1789702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1789980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1790115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1790300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgolnp5il 2022-12-01T10:40:43.1790482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgolnp5il/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1790755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1790880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1791161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1791296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1791480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_obrqpq 2022-12-01T10:40:43.1791673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_obrqpq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1791947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1792070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1792334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1792473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1792658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2llw53b 2022-12-01T10:40:43.1792850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2llw53b/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1793014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1793179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1793340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1793501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1793677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.1793842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.1794144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1794318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.1794488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.1794813Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1795111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1795397Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.1795667Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1795963Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1796228Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1796480Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1796739Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1796998Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1797070Z ok (2.452s) 2022-12-01T10:40:43.1797076Z 2022-12-01T10:40:43.1797273Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1797348Z Ran 1 test in 2.453s 2022-12-01T10:40:43.1797354Z 2022-12-01T10:40:43.1797413Z OK 2022-12-01T10:40:43.1797418Z 2022-12-01T10:40:43.1797501Z Generating XML reports... 2022-12-01T10:40:43.1797896Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103817.xml 2022-12-01T10:40:43.1798171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1798406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1798689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1798827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1799014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp2xsywr 2022-12-01T10:40:43.1799214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp2xsywr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1799471Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1799476Z 2022-12-01T10:40:43.1799551Z Running tests... 2022-12-01T10:40:43.1799749Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1800010Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 527 2022-12-01T10:40:43.1800153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 528 2022-12-01T10:40:43.1800310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 529 2022-12-01T10:40:43.1800461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 530 2022-12-01T10:40:43.1800738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1800863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1801187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1801323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1801510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyonce753 2022-12-01T10:40:43.1801693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyonce753/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1801968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1802130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1802410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1802544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1802731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2fz5gdk 2022-12-01T10:40:43.1802930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2fz5gdk/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1803200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1803323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1803589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1803723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1803904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpej_h_3bc 2022-12-01T10:40:43.1804094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpej_h_3bc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1804366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1804490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1804767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1804899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1805070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvywqaexf 2022-12-01T10:40:43.1805267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvywqaexf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1805429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1805593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1805756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1805918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1806185Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1806451Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1806717Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1806785Z ok (2.354s) 2022-12-01T10:40:43.1806790Z 2022-12-01T10:40:43.1806976Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1807051Z Ran 1 test in 2.354s 2022-12-01T10:40:43.1807056Z 2022-12-01T10:40:43.1807117Z OK 2022-12-01T10:40:43.1807122Z 2022-12-01T10:40:43.1807233Z Generating XML reports... 2022-12-01T10:40:43.1807624Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103821.xml 2022-12-01T10:40:43.1807962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1808092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1808406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1808542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1808715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpple2idpg 2022-12-01T10:40:43.1808912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpple2idpg/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1809173Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1809178Z 2022-12-01T10:40:43.1809253Z Running tests... 2022-12-01T10:40:43.1809451Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1809716Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 752 2022-12-01T10:40:43.1809872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 753 2022-12-01T10:40:43.1810029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 754 2022-12-01T10:40:43.1810168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 755 2022-12-01T10:40:43.1810446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1810576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1810863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1811000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1811184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3x13w00j 2022-12-01T10:40:43.1811378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3x13w00j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1811652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1811775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1812043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1812176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1812361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprewl4900 2022-12-01T10:40:43.1812558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprewl4900/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1812827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1812951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1813229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1813365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1813535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uiytou6 2022-12-01T10:40:43.1813730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uiytou6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1814029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1814153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1814433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1814568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1814756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx6bchuu 2022-12-01T10:40:43.1814977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx6bchuu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1815140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1815291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1815453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1815612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1820134Z ok (1.840s) 2022-12-01T10:40:43.1820144Z 2022-12-01T10:40:43.1820385Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1820472Z Ran 1 test in 1.841s 2022-12-01T10:40:43.1820478Z 2022-12-01T10:40:43.1820539Z OK 2022-12-01T10:40:43.1820544Z 2022-12-01T10:40:43.1820633Z Generating XML reports... 2022-12-01T10:40:43.1821047Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103825.xml 2022-12-01T10:40:43.1821330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1821444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1821728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1821869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1822058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72eqz3y6 2022-12-01T10:40:43.1822255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72eqz3y6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1822514Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1822521Z 2022-12-01T10:40:43.1822594Z Running tests... 2022-12-01T10:40:43.1822793Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1823048Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 807 2022-12-01T10:40:43.1823204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 808 2022-12-01T10:40:43.1823358Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 809 2022-12-01T10:40:43.1823514Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 810 2022-12-01T10:40:43.1823791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1823916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1824201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1824340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1824527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd39yvecr 2022-12-01T10:40:43.1824712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd39yvecr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1825073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1825199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1825482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1825619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1825806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5wlda70 2022-12-01T10:40:43.1826111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1826235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1826500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1826636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1826823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bpzuuqu 2022-12-01T10:40:43.1827019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5wlda70/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1827212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bpzuuqu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1827484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1827609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1827888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1828022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1828191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasolj_xy 2022-12-01T10:40:43.1828387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasolj_xy/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1828551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1828716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1828876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1829036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1829108Z ok (2.343s) 2022-12-01T10:40:43.1829113Z 2022-12-01T10:40:43.1829312Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1829375Z Ran 1 test in 2.343s 2022-12-01T10:40:43.1829392Z 2022-12-01T10:40:43.1829440Z OK 2022-12-01T10:40:43.1829444Z 2022-12-01T10:40:43.1829531Z Generating XML reports... 2022-12-01T10:40:43.1829926Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103828.xml 2022-12-01T10:40:43.1830202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1830326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1830606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1830742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1830931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83vvar72 2022-12-01T10:40:43.1831116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83vvar72/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1831371Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1831376Z 2022-12-01T10:40:43.1831451Z Running tests... 2022-12-01T10:40:43.1831683Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1831947Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1042 2022-12-01T10:40:43.1832103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1043 2022-12-01T10:40:43.1832255Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1044 2022-12-01T10:40:43.1832440Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1045 2022-12-01T10:40:43.1832714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1832825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1833106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1833242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1833429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6d4bvg_ 2022-12-01T10:40:43.1833625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6d4bvg_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1833896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1834018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1834300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1834421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1834604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq42nyh46 2022-12-01T10:40:43.1834798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq42nyh46/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1835070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1835192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1835467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1835602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1835787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh8qw8ay 2022-12-01T10:40:43.1835980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh8qw8ay/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1836238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1836360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1836640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1836774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1836956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxuwal_ie 2022-12-01T10:40:43.1837148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxuwal_ie/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1837312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1837480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1837627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1837789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1837855Z ok (2.243s) 2022-12-01T10:40:43.1837861Z 2022-12-01T10:40:43.1838086Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1838165Z Ran 1 test in 2.244s 2022-12-01T10:40:43.1838170Z 2022-12-01T10:40:43.1838406Z OK 2022-12-01T10:40:43.1838413Z 2022-12-01T10:40:43.1838541Z Generating XML reports... 2022-12-01T10:40:43.1838992Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103832.xml 2022-12-01T10:40:43.1839333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1839446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1839727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1839862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1840050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0o4uykb 2022-12-01T10:40:43.1840246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0o4uykb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1840502Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1840508Z 2022-12-01T10:40:43.1840582Z Running tests... 2022-12-01T10:40:43.1840778Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1841034Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1277 2022-12-01T10:40:43.1841178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1278 2022-12-01T10:40:43.1841332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1279 2022-12-01T10:40:43.1841483Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1280 2022-12-01T10:40:43.1841761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1841885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1842169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1842304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1842491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7n6dcz13 2022-12-01T10:40:43.1842676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7n6dcz13/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1842948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1843071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1843353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1843488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1843673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqk8b9722 2022-12-01T10:40:43.1843866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqk8b9722/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1844135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1844261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1844526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1844662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1844845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxvwfn3w 2022-12-01T10:40:43.1845081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxvwfn3w/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1845353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1845476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1845755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1845921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1846092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhoeomvu 2022-12-01T10:40:43.1846288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhoeomvu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1846452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1846619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1846783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1846943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1847011Z ok (2.446s) 2022-12-01T10:40:43.1847017Z 2022-12-01T10:40:43.1847218Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1847296Z Ran 1 test in 2.446s 2022-12-01T10:40:43.1847302Z 2022-12-01T10:40:43.1847350Z OK 2022-12-01T10:40:43.1847355Z 2022-12-01T10:40:43.1847438Z Generating XML reports... 2022-12-01T10:40:43.1847897Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103836.xml 2022-12-01T10:40:43.1848174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1848301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1848583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1848717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1848904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp728k4hyi 2022-12-01T10:40:43.1849088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp728k4hyi/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1849350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1849356Z 2022-12-01T10:40:43.1849429Z Running tests... 2022-12-01T10:40:43.1849626Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1849894Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1512 2022-12-01T10:40:43.1850051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1513 2022-12-01T10:40:43.1850208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1514 2022-12-01T10:40:43.1850361Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1515 2022-12-01T10:40:43.1850635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1850748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1851034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1851168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1851353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ldjv47h 2022-12-01T10:40:43.1851583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ldjv47h/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1851856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1851978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1852258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1852420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1852589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9t49e59o 2022-12-01T10:40:43.1852782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9t49e59o/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1853051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1853177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1853456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1853589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1853772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmtbbvpl 2022-12-01T10:40:43.1853967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmtbbvpl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1854227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1854349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1854628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1854760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1854945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8w1po2ri 2022-12-01T10:40:43.1855138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8w1po2ri/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1855301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1855464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1855627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1855778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1855844Z ok (2.345s) 2022-12-01T10:40:43.1855849Z 2022-12-01T10:40:43.1856045Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1856120Z Ran 1 test in 2.345s 2022-12-01T10:40:43.1856125Z 2022-12-01T10:40:43.1856184Z OK 2022-12-01T10:40:43.1856191Z 2022-12-01T10:40:43.1856274Z Generating XML reports... 2022-12-01T10:40:43.1856666Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103840.xml 2022-12-01T10:40:43.1856942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1857054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1857339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1857474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1857660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp367ltyee 2022-12-01T10:40:43.1857857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp367ltyee/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1858142Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1858148Z 2022-12-01T10:40:43.1858221Z Running tests... 2022-12-01T10:40:43.1858417Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1858671Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1771 2022-12-01T10:40:43.1858857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1772 2022-12-01T10:40:43.1859010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1773 2022-12-01T10:40:43.1859161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1774 2022-12-01T10:40:43.1859438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1859561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1859846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1859980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1860165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmkap52q 2022-12-01T10:40:43.1860349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmkap52q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1860623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1860747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1861024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1861156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1861342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpossd_kw0 2022-12-01T10:40:43.1861611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1861733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1862010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1862134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1862317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsiawoop 2022-12-01T10:40:43.1862508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpossd_kw0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1862699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsiawoop/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1862970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1863093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1863371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1863502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1863682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_akdrrwx 2022-12-01T10:40:43.1863863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_akdrrwx/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1864026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1864189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1864352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1864543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1864611Z ok (2.148s) 2022-12-01T10:40:43.1864616Z 2022-12-01T10:40:43.1864814Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1864890Z Ran 1 test in 2.148s 2022-12-01T10:40:43.1864895Z 2022-12-01T10:40:43.1864942Z OK 2022-12-01T10:40:43.1864947Z 2022-12-01T10:40:43.1865029Z Generating XML reports... 2022-12-01T10:40:43.1865449Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103843.xml 2022-12-01T10:40:43.1865722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1865846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1866130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1866264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1866450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k6it5yq 2022-12-01T10:40:43.1866645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k6it5yq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1866887Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1866895Z 2022-12-01T10:40:43.1866967Z Running tests... 2022-12-01T10:40:43.1867164Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1867417Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2018 2022-12-01T10:40:43.1867575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2019 2022-12-01T10:40:43.1867729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2020 2022-12-01T10:40:43.1867880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2021 2022-12-01T10:40:43.1868153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1868264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1868543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1868681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1868867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewp083us 2022-12-01T10:40:43.1869060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewp083us/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1869334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1869457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1869736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1869869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1870041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzdpa7b7 2022-12-01T10:40:43.1870236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzdpa7b7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1870505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1870626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1870906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1871076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1871260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfw0pwpj5 2022-12-01T10:40:43.1871453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfw0pwpj5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1871708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1871857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1872134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1872266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1872449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptju4m0i4 2022-12-01T10:40:43.1872642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptju4m0i4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1872805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1872972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1873133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1873283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1873354Z ok (2.248s) 2022-12-01T10:40:43.1873359Z 2022-12-01T10:40:43.1873556Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1873631Z Ran 1 test in 2.248s 2022-12-01T10:40:43.1873636Z 2022-12-01T10:40:43.1873695Z OK 2022-12-01T10:40:43.1873700Z 2022-12-01T10:40:43.1873783Z Generating XML reports... 2022-12-01T10:40:43.1874173Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103847.xml 2022-12-01T10:40:43.1874450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1874562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1874844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1874980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1875169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9j699wu 2022-12-01T10:40:43.1875363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9j699wu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1875619Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1875625Z 2022-12-01T10:40:43.1875698Z Running tests... 2022-12-01T10:40:43.1875895Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1876150Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2265 2022-12-01T10:40:43.1876294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2266 2022-12-01T10:40:43.1876448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2267 2022-12-01T10:40:43.1876604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2268 2022-12-01T10:40:43.1876879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1877003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1877284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1877449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1877638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzhqftzy 2022-12-01T10:40:43.1877837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzhqftzy/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1878097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1878363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1878704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1878841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1879028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai1m_v5h 2022-12-01T10:40:43.1879223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai1m_v5h/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1879497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1879621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1879886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1880022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1880213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pqnu9eq 2022-12-01T10:40:43.1880408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pqnu9eq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1880682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1880805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1881089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1881224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1881410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmdd1ou8 2022-12-01T10:40:43.1881591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmdd1ou8/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1881757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1881919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1882081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1882241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1882308Z ok (2.244s) 2022-12-01T10:40:43.1882313Z 2022-12-01T10:40:43.1882511Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1882586Z Ran 1 test in 2.245s 2022-12-01T10:40:43.1882591Z 2022-12-01T10:40:43.1882638Z OK 2022-12-01T10:40:43.1882644Z 2022-12-01T10:40:43.1882727Z Generating XML reports... 2022-12-01T10:40:43.1883116Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103851.xml 2022-12-01T10:40:43.1883390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1883517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1883794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1883927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1884155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7jh9fhj 2022-12-01T10:40:43.1884353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7jh9fhj/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1884597Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1884602Z 2022-12-01T10:40:43.1884676Z Running tests... 2022-12-01T10:40:43.1884872Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1885163Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2512 2022-12-01T10:40:43.1885318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2513 2022-12-01T10:40:43.1885471Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2514 2022-12-01T10:40:43.1885621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2515 2022-12-01T10:40:43.1885895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1886006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1886292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1886428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1886617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd27kl8p2 2022-12-01T10:40:43.1886816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd27kl8p2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1887089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1887213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1887495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1887630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1887857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp634hyn2l 2022-12-01T10:40:43.1888055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp634hyn2l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1888328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1888454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1888732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1888866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1889055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7rtteuc 2022-12-01T10:40:43.1889253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7rtteuc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1889526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1889636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1889918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1890056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1890243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo1spc39 2022-12-01T10:40:43.1890438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo1spc39/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1890603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1890803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1890965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1891114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1891182Z ok (2.244s) 2022-12-01T10:40:43.1891187Z 2022-12-01T10:40:43.1891386Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1891489Z Ran 1 test in 2.244s 2022-12-01T10:40:43.1891494Z 2022-12-01T10:40:43.1891553Z OK 2022-12-01T10:40:43.1891557Z 2022-12-01T10:40:43.1891640Z Generating XML reports... 2022-12-01T10:40:43.1892032Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103855.xml 2022-12-01T10:40:43.1892304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1892429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1892696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1892829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1893011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyw6al7l 2022-12-01T10:40:43.1893209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyw6al7l/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1893468Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1893474Z 2022-12-01T10:40:43.1893546Z Running tests... 2022-12-01T10:40:43.1893742Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1894001Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2939 2022-12-01T10:40:43.1894145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2940 2022-12-01T10:40:43.1894299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2941 2022-12-01T10:40:43.1894451Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2942 2022-12-01T10:40:43.1894723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1894850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1895130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1895265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1895450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpba01m5br 2022-12-01T10:40:43.1895720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1895830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1896109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1896241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1896425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpno7ye8gy 2022-12-01T10:40:43.1896622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpba01m5br/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1896811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpno7ye8gy/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1897081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1897230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1897495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1897628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1897812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkyul11j 2022-12-01T10:40:43.1898007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkyul11j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1898312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1898438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1898719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1898853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1899042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc63idgh9 2022-12-01T10:40:43.1899225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc63idgh9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1899389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1899554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1899718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1899878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1899944Z ok (2.143s) 2022-12-01T10:40:43.1899950Z 2022-12-01T10:40:43.1900148Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1900221Z Ran 1 test in 2.144s 2022-12-01T10:40:43.1900226Z 2022-12-01T10:40:43.1900274Z OK 2022-12-01T10:40:43.1900292Z 2022-12-01T10:40:43.1900364Z Generating XML reports... 2022-12-01T10:40:43.1900757Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103858.xml 2022-12-01T10:40:43.1901029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1901152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1901433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1901566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1901752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsrnbk23 2022-12-01T10:40:43.1901949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsrnbk23/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1902191Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1902209Z 2022-12-01T10:40:43.1902269Z Running tests... 2022-12-01T10:40:43.1902466Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1902724Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3162 2022-12-01T10:40:43.1902880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3163 2022-12-01T10:40:43.1903032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3164 2022-12-01T10:40:43.1903184Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3165 2022-12-01T10:40:43.1903458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1903582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1903880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1904016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1904204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxli3d51 2022-12-01T10:40:43.1904401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxli3d51/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1904702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1904825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1905103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1905238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1905411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nraxe94 2022-12-01T10:40:43.1905605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nraxe94/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1905874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1905998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1906275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1906411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1906594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtxifhpw 2022-12-01T10:40:43.1906792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtxifhpw/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1907067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1907177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1907453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1907588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1907774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0ycl01p 2022-12-01T10:40:43.1907971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0ycl01p/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1908134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1908297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1908459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1908609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1908676Z ok (2.243s) 2022-12-01T10:40:43.1908681Z 2022-12-01T10:40:43.1908876Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1908955Z Ran 1 test in 2.244s 2022-12-01T10:40:43.1908961Z 2022-12-01T10:40:43.1909019Z OK 2022-12-01T10:40:43.1909024Z 2022-12-01T10:40:43.1909108Z Generating XML reports... 2022-12-01T10:40:43.1909504Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103902.xml 2022-12-01T10:40:43.1909776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1909899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1910193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1910328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1910515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzkaj7rt 2022-12-01T10:40:43.1910710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzkaj7rt/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1910964Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1910996Z 2022-12-01T10:40:43.1911070Z Running tests... 2022-12-01T10:40:43.1911265Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1911524Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3397 2022-12-01T10:40:43.1911680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3398 2022-12-01T10:40:43.1911824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3399 2022-12-01T10:40:43.1911976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3400 2022-12-01T10:40:43.1912252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1912374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1912655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1912791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1912976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8eopay5n 2022-12-01T10:40:43.1913171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8eopay5n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1913431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1913553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1913833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1913967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1914151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4ajgf1e 2022-12-01T10:40:43.1914347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4ajgf1e/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1914615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1914737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1915014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1915137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1915319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq54c7x3f 2022-12-01T10:40:43.1915510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq54c7x3f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1915779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1915903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1916182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1916314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1916496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzi_rel6 2022-12-01T10:40:43.1916714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzi_rel6/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1916880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1917043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1917204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1917364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1917457Z ok (2.144s) 2022-12-01T10:40:43.1917463Z 2022-12-01T10:40:43.1917661Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1917735Z Ran 1 test in 2.144s 2022-12-01T10:40:43.1917740Z 2022-12-01T10:40:43.1917798Z OK 2022-12-01T10:40:43.1917803Z 2022-12-01T10:40:43.1917874Z Generating XML reports... 2022-12-01T10:40:43.1918398Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103906.xml 2022-12-01T10:40:43.1918674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1918800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1919084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1919220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1919407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3yn3yyq7 2022-12-01T10:40:43.1919603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3yn3yyq7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1919845Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1919865Z 2022-12-01T10:40:43.1919924Z Running tests... 2022-12-01T10:40:43.1920122Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1920387Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3632 2022-12-01T10:40:43.1920543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3633 2022-12-01T10:40:43.1920696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3634 2022-12-01T10:40:43.1920850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3635 2022-12-01T10:40:43.1921125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1921250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1921520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1921657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1921844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8f8jc3j 2022-12-01T10:40:43.1922041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8f8jc3j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1922310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1922432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1922715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1922849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1923033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfi9jad4t 2022-12-01T10:40:43.1923216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfi9jad4t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1923532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1923655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1923934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1924070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1924288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbe84uht 2022-12-01T10:40:43.1924556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1924680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1924945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1925082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1925266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptaha7zrb 2022-12-01T10:40:43.1925461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbe84uht/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1925654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptaha7zrb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1925820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1925982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1926143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1926303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1926358Z ok (2.343s) 2022-12-01T10:40:43.1926363Z 2022-12-01T10:40:43.1926561Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1926636Z Ran 1 test in 2.343s 2022-12-01T10:40:43.1926641Z 2022-12-01T10:40:43.1926703Z OK 2022-12-01T10:40:43.1926708Z 2022-12-01T10:40:43.1926791Z Generating XML reports... 2022-12-01T10:40:43.1927185Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103909.xml 2022-12-01T10:40:43.1927459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1927584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1927903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1928040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1928226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1o24l6j 2022-12-01T10:40:43.1928421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1o24l6j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1928679Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1928684Z 2022-12-01T10:40:43.1928759Z Running tests... 2022-12-01T10:40:43.1928954Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1929229Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3867 2022-12-01T10:40:43.1929386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3868 2022-12-01T10:40:43.1929527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3869 2022-12-01T10:40:43.1929681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3870 2022-12-01T10:40:43.1929986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1930110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1930388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1930522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1930737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptac8ehdg 2022-12-01T10:40:43.1930934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptac8ehdg/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1931192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1931313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1931592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1931726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1931914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphreh26f 2022-12-01T10:40:43.1932110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphreh26f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1932378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1932503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1932781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1932902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1933086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9lbyl9n 2022-12-01T10:40:43.1933281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9lbyl9n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1933548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1933669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1933944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1934078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1934261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmvrzyje 2022-12-01T10:40:43.1934455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmvrzyje/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1934606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1934771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1934933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1935096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1935164Z ok (2.146s) 2022-12-01T10:40:43.1935169Z 2022-12-01T10:40:43.1935368Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1935444Z Ran 1 test in 2.146s 2022-12-01T10:40:43.1935449Z 2022-12-01T10:40:43.1935509Z OK 2022-12-01T10:40:43.1935513Z 2022-12-01T10:40:43.1935584Z Generating XML reports... 2022-12-01T10:40:43.1935977Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103913.xml 2022-12-01T10:40:43.1936252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1936413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1936695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1936831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1937016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8y_mrz_i 2022-12-01T10:40:43.1937237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8y_mrz_i/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1937494Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1937500Z 2022-12-01T10:40:43.1937560Z Running tests... 2022-12-01T10:40:43.1937759Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1938028Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4102 2022-12-01T10:40:43.1938186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4103 2022-12-01T10:40:43.1938340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4104 2022-12-01T10:40:43.1938492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4105 2022-12-01T10:40:43.1938768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1938896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1939166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1939303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1939491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_rquqdr 2022-12-01T10:40:43.1939692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_rquqdr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1939967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1940091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1940371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1940511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1940697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphudu11kp 2022-12-01T10:40:43.1940880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphudu11kp/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1941155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1941280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1941559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1941692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1941876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnvne8re 2022-12-01T10:40:43.1942069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnvne8re/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1942341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1942451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1942728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1942888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1943076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiiipvz2f 2022-12-01T10:40:43.1943270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiiipvz2f/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1943434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1943597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1943787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1943948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1944004Z ok (4.247s) 2022-12-01T10:40:43.1944009Z 2022-12-01T10:40:43.1944208Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1944282Z Ran 1 test in 4.247s 2022-12-01T10:40:43.1944287Z 2022-12-01T10:40:43.1944348Z OK 2022-12-01T10:40:43.1944355Z 2022-12-01T10:40:43.1944440Z Generating XML reports... 2022-12-01T10:40:43.1944835Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103917.xml 2022-12-01T10:40:43.1945108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1945232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1945514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1945637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1945821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdsir9y9 2022-12-01T10:40:43.1946016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdsir9y9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1946275Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1946281Z 2022-12-01T10:40:43.1946356Z Running tests... 2022-12-01T10:40:43.1946556Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1946824Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4325 2022-12-01T10:40:43.1946983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4326 2022-12-01T10:40:43.1947124Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4327 2022-12-01T10:40:43.1947277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4328 2022-12-01T10:40:43.1947550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1947677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1947957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1948091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1948276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8dj4che 2022-12-01T10:40:43.1948472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8dj4che/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1948749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1948858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1949138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1949272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1949487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20m350ht 2022-12-01T10:40:43.1949684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20m350ht/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1949955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1950080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1950389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1950512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1950695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp757vzb6n 2022-12-01T10:40:43.1950887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp757vzb6n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1951160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1951282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1951563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1951695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1951881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1z9fz5t 2022-12-01T10:40:43.1952075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1z9fz5t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1952226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1952389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1952551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1952713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1952782Z ok (4.150s) 2022-12-01T10:40:43.1952787Z 2022-12-01T10:40:43.1952984Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1953058Z Ran 1 test in 4.151s 2022-12-01T10:40:43.1953064Z 2022-12-01T10:40:43.1953123Z OK 2022-12-01T10:40:43.1953128Z 2022-12-01T10:40:43.1953201Z Generating XML reports... 2022-12-01T10:40:43.1953594Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103922.xml 2022-12-01T10:40:43.1953866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1953992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1954275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1954410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1954595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal1z6w33 2022-12-01T10:40:43.1954789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal1z6w33/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1955045Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1955053Z 2022-12-01T10:40:43.1955113Z Running tests... 2022-12-01T10:40:43.1955310Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1955563Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4548 2022-12-01T10:40:43.1955719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4549 2022-12-01T10:40:43.1955899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4550 2022-12-01T10:40:43.1956051Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4551 2022-12-01T10:40:43.1956326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1956449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1956731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1956891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1957076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5etmttek 2022-12-01T10:40:43.1957272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5etmttek/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1957548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1957671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1957950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1958083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1958398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicx8vqmm 2022-12-01T10:40:43.1958585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicx8vqmm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1958860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1958985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1959267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1959403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1959589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjzldbxm 2022-12-01T10:40:43.1959785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjzldbxm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1960058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1960184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1960451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1960587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1960770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88i5coet 2022-12-01T10:40:43.1960967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88i5coet/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1961131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1961299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1961465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1961629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1961893Z [W tensorpipe_agent.cpp:940] RPC agent for worker2 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1962163Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.1962528Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #30 to worker0, as the agent is shutting down 2022-12-01T10:40:43.1962811Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #28 to worker3, as the agent is shutting down 2022-12-01T10:40:43.1963093Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #28 to worker2, as the agent is shutting down 2022-12-01T10:40:43.1963408Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #30 to worker1, as the agent is shutting down 2022-12-01T10:40:43.1963477Z ok (15.167s) 2022-12-01T10:40:43.1963482Z 2022-12-01T10:40:43.1963681Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1963758Z Ran 1 test in 15.167s 2022-12-01T10:40:43.1963763Z 2022-12-01T10:40:43.1963824Z OK 2022-12-01T10:40:43.1963829Z 2022-12-01T10:40:43.1963900Z Generating XML reports... 2022-12-01T10:40:43.1964295Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103928.xml 2022-12-01T10:40:43.1964571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1964697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1964979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1965118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1965304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6une0j0t 2022-12-01T10:40:43.1965499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6une0j0t/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1965754Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1965760Z 2022-12-01T10:40:43.1965822Z Running tests... 2022-12-01T10:40:43.1966022Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1966279Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4771 2022-12-01T10:40:43.1966435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4772 2022-12-01T10:40:43.1966588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4773 2022-12-01T10:40:43.1966743Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4774 2022-12-01T10:40:43.1967018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1967140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1967410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1967546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1967785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxq7pnu54 2022-12-01T10:40:43.1967984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxq7pnu54/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1968262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1968388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1968671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1968808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1968995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9t4zuph 2022-12-01T10:40:43.1969212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9t4zuph/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1969484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1969607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1969886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1970049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1970233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5hswh82 2022-12-01T10:40:43.1970427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5hswh82/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1970698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1970823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1971088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1971221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1971405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnq3f7p0n 2022-12-01T10:40:43.1971599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnq3f7p0n/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1971765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1971928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1972089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1972249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1972305Z ok (2.344s) 2022-12-01T10:40:43.1972313Z 2022-12-01T10:40:43.1972514Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1972586Z Ran 1 test in 2.344s 2022-12-01T10:40:43.1972592Z 2022-12-01T10:40:43.1972651Z OK 2022-12-01T10:40:43.1972655Z 2022-12-01T10:40:43.1972738Z Generating XML reports... 2022-12-01T10:40:43.1973130Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103945.xml 2022-12-01T10:40:43.1973405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1973530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1973812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1973933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1974121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb35nlcoo 2022-12-01T10:40:43.1974313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb35nlcoo/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1974569Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1974574Z 2022-12-01T10:40:43.1974647Z Running tests... 2022-12-01T10:40:43.1974845Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1975116Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5006 2022-12-01T10:40:43.1975270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5007 2022-12-01T10:40:43.1975411Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5008 2022-12-01T10:40:43.1975593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5009 2022-12-01T10:40:43.1975869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1975992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1976274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1976437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1976623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp445fcj6s 2022-12-01T10:40:43.1976815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp445fcj6s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1977086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1977198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1977476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1977609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1977791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hb6f5o0 2022-12-01T10:40:43.1977985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hb6f5o0/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1978258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1978380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1978659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1978779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1978963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4pl7qpr 2022-12-01T10:40:43.1979156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4pl7qpr/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1979424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1979546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1979829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1979962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1980146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1gkbt2k 2022-12-01T10:40:43.1980338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1gkbt2k/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1980490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1980653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1980814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1980974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1981042Z ok (2.242s) 2022-12-01T10:40:43.1981049Z 2022-12-01T10:40:43.1981247Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1981322Z Ran 1 test in 2.243s 2022-12-01T10:40:43.1981327Z 2022-12-01T10:40:43.1981386Z OK 2022-12-01T10:40:43.1981390Z 2022-12-01T10:40:43.1981461Z Generating XML reports... 2022-12-01T10:40:43.1981853Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103948.xml 2022-12-01T10:40:43.1982155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1982281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1982564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1982700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1982885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuiqv1vot 2022-12-01T10:40:43.1983109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuiqv1vot/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1983366Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1983371Z 2022-12-01T10:40:43.1983431Z Running tests... 2022-12-01T10:40:43.1983626Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1983895Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5241 2022-12-01T10:40:43.1984051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5242 2022-12-01T10:40:43.1984203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5243 2022-12-01T10:40:43.1984354Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5244 2022-12-01T10:40:43.1984629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1984752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1985029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1985152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1985338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwsy_mhl 2022-12-01T10:40:43.1985532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwsy_mhl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1985803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1985926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1986202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1986338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1986519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6_a9jr9 2022-12-01T10:40:43.1986697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6_a9jr9/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1986972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1987095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1987372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1987505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1987686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21tugrsl 2022-12-01T10:40:43.1987884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21tugrsl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1988154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1988275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1988571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1988705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1988886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01vbsh61 2022-12-01T10:40:43.1989077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01vbsh61/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1989240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1989431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1989635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1989795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1989850Z ok (2.344s) 2022-12-01T10:40:43.1989867Z 2022-12-01T10:40:43.1990052Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1990128Z Ran 1 test in 2.344s 2022-12-01T10:40:43.1990132Z 2022-12-01T10:40:43.1990191Z OK 2022-12-01T10:40:43.1990196Z 2022-12-01T10:40:43.1990279Z Generating XML reports... 2022-12-01T10:40:43.1990670Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103952.xml 2022-12-01T10:40:43.1990944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1991069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1991351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1991474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1991658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso8d4ox2 2022-12-01T10:40:43.1991857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso8d4ox2/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1992112Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.1992117Z 2022-12-01T10:40:43.1992189Z Running tests... 2022-12-01T10:40:43.1992384Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1992664Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5476 2022-12-01T10:40:43.1992822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5477 2022-12-01T10:40:43.1992975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5478 2022-12-01T10:40:43.1993115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5479 2022-12-01T10:40:43.1993394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1993516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1993799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1993933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1994118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ejvqe66 2022-12-01T10:40:43.1994317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ejvqe66/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1994588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1994698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1994977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1995138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1995323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppuzvua29 2022-12-01T10:40:43.1995517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppuzvua29/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1995786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1995938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1996217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1996352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1996522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4w1eb1s 2022-12-01T10:40:43.1996718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4w1eb1s/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1996987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1997109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.1997385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.1997519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.1997700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23ky6oji 2022-12-01T10:40:43.1997892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23ky6oji/_remote_module_non_scriptable.py 2022-12-01T10:40:43.1998043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.1998329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.1998494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.1998660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.1998729Z ok (2.146s) 2022-12-01T10:40:43.1998734Z 2022-12-01T10:40:43.1998933Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.1999010Z Ran 1 test in 2.146s 2022-12-01T10:40:43.1999017Z 2022-12-01T10:40:43.1999078Z OK 2022-12-01T10:40:43.1999083Z 2022-12-01T10:40:43.1999168Z Generating XML reports... 2022-12-01T10:40:43.1999548Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103956.xml 2022-12-01T10:40:43.1999822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.1999946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2000230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2000367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2000555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyol2ex54 2022-12-01T10:40:43.2000752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyol2ex54/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2001012Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2001017Z 2022-12-01T10:40:43.2001091Z Running tests... 2022-12-01T10:40:43.2001277Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2001551Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5711 2022-12-01T10:40:43.2001753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5712 2022-12-01T10:40:43.2001911Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5713 2022-12-01T10:40:43.2002062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5714 2022-12-01T10:40:43.2002338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2002496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2002778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2002901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2003087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0es4f0vq 2022-12-01T10:40:43.2003285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0es4f0vq/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2003559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2003682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2003961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2004096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2004285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwcd6ph7 2022-12-01T10:40:43.2004479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwcd6ph7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2004737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2004857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2005138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2005271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2005455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8o2gubc 2022-12-01T10:40:43.2005647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8o2gubc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2005919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2006039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2006302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2006435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2006619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9elhhsm_ 2022-12-01T10:40:43.2006815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9elhhsm_/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2006978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2007140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2007302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2007465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2007532Z ok (2.143s) 2022-12-01T10:40:43.2007538Z 2022-12-01T10:40:43.2007773Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2007854Z Ran 1 test in 2.144s 2022-12-01T10:40:43.2007858Z 2022-12-01T10:40:43.2007918Z OK 2022-12-01T10:40:43.2007924Z 2022-12-01T10:40:43.2008054Z Generating XML reports... 2022-12-01T10:40:43.2008454Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104000.xml 2022-12-01T10:40:43.2008729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2008853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2009135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2009288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2009475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgaxlphfn 2022-12-01T10:40:43.2009673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgaxlphfn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2009931Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2009938Z 2022-12-01T10:40:43.2010010Z Running tests... 2022-12-01T10:40:43.2010209Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2010465Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5946 2022-12-01T10:40:43.2010621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5947 2022-12-01T10:40:43.2010778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5948 2022-12-01T10:40:43.2010916Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5949 2022-12-01T10:40:43.2011190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2011315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2011601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2011737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2011924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6xgqe3v 2022-12-01T10:40:43.2012118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6xgqe3v/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2012393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2012519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2012788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2012920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2013107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulol3q1q 2022-12-01T10:40:43.2013380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2013501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2013782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2013916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2014102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrl37abk 2022-12-01T10:40:43.2014285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulol3q1q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2014473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrl37abk/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2014773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2014896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2015176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2015309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2015494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt72jrobv 2022-12-01T10:40:43.2015713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt72jrobv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2015877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2016028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2016191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2016353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2016420Z ok (2.152s) 2022-12-01T10:40:43.2016425Z 2022-12-01T10:40:43.2016623Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2016697Z Ran 1 test in 2.152s 2022-12-01T10:40:43.2016702Z 2022-12-01T10:40:43.2016762Z OK 2022-12-01T10:40:43.2016766Z 2022-12-01T10:40:43.2016849Z Generating XML reports... 2022-12-01T10:40:43.2017226Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104003.xml 2022-12-01T10:40:43.2017503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2017627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2017906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2018042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2018227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7982kdqo 2022-12-01T10:40:43.2018421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7982kdqo/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2018678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2018685Z 2022-12-01T10:40:43.2018757Z Running tests... 2022-12-01T10:40:43.2018943Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2019207Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6181 2022-12-01T10:40:43.2019360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6182 2022-12-01T10:40:43.2019514Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6183 2022-12-01T10:40:43.2019667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6184 2022-12-01T10:40:43.2019941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2020064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2020347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2020472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2020659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr1v6hbf 2022-12-01T10:40:43.2020856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr1v6hbf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2021124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2021273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2021551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2021684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2021868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_vg2awm 2022-12-01T10:40:43.2022089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_vg2awm/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2022347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2022469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2022748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2022883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2023066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuzt35uf 2022-12-01T10:40:43.2023261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuzt35uf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2023530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2023652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2023931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2024052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2024234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpth9rok7q 2022-12-01T10:40:43.2024428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpth9rok7q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2024592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2024756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2024917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2025078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2025147Z ok (2.145s) 2022-12-01T10:40:43.2025152Z 2022-12-01T10:40:43.2025339Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2025414Z Ran 1 test in 2.146s 2022-12-01T10:40:43.2025419Z 2022-12-01T10:40:43.2025479Z OK 2022-12-01T10:40:43.2025484Z 2022-12-01T10:40:43.2025567Z Generating XML reports... 2022-12-01T10:40:43.2025960Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104007.xml 2022-12-01T10:40:43.2026234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2026358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2026638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2026772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2026947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevngg7h5 2022-12-01T10:40:43.2027142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevngg7h5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2027394Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2027399Z 2022-12-01T10:40:43.2027471Z Running tests... 2022-12-01T10:40:43.2027694Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2027958Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6448 2022-12-01T10:40:43.2028114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6449 2022-12-01T10:40:43.2028266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6450 2022-12-01T10:40:43.2028404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6451 2022-12-01T10:40:43.2028706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2028831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2029112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2029246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2029433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3bzm6jb 2022-12-01T10:40:43.2029628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3bzm6jb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2029898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2030020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2030288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2030421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2030605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkqnmsj5 2022-12-01T10:40:43.2030802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkqnmsj5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2031072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2031194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2031472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2031604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2031774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74b7zawc 2022-12-01T10:40:43.2031966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74b7zawc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2032237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2032361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2032641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2032774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2032957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8nhaa4v 2022-12-01T10:40:43.2033150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8nhaa4v/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2033313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2033466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2033628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2033790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2033904Z Rank 0 finished testing 20 times in 0.09405517578125 seconds. 2022-12-01T10:40:43.2034045Z Rank 1 finished testing 20 times in 0.09088516235351562 seconds. 2022-12-01T10:40:43.2034160Z Rank 2 finished testing 20 times in 0.14188933372497559 seconds. 2022-12-01T10:40:43.2034272Z Rank 3 finished testing 20 times in 0.1129922866821289 seconds. 2022-12-01T10:40:43.2034339Z ok (2.248s) 2022-12-01T10:40:43.2034344Z 2022-12-01T10:40:43.2034528Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2034603Z Ran 1 test in 2.248s 2022-12-01T10:40:43.2034634Z 2022-12-01T10:40:43.2034694Z OK 2022-12-01T10:40:43.2034699Z 2022-12-01T10:40:43.2034782Z Generating XML reports... 2022-12-01T10:40:43.2035178Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104011.xml 2022-12-01T10:40:43.2035451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2035575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2035853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2035988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2036162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjv1m3pl 2022-12-01T10:40:43.2036356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjv1m3pl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2036616Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2036622Z 2022-12-01T10:40:43.2036693Z Running tests... 2022-12-01T10:40:43.2036889Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2037171Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6683 2022-12-01T10:40:43.2037330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6684 2022-12-01T10:40:43.2037482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6685 2022-12-01T10:40:43.2037621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6686 2022-12-01T10:40:43.2037893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2038017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2038442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2038579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2038766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsu9h0zu 2022-12-01T10:40:43.2038965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsu9h0zu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2039239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2039361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2039626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2039762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2039950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyvl1huy 2022-12-01T10:40:43.2040145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyvl1huy/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2040414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2040536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2040872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2041007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2041178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8n226z4 2022-12-01T10:40:43.2041371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8n226z4/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2041675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2041797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2042074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2042208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2042395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpri2e8xgn 2022-12-01T10:40:43.2042588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpri2e8xgn/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2042753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2042905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2043067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2043232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2043302Z ok (1.944s) 2022-12-01T10:40:43.2043307Z 2022-12-01T10:40:43.2043506Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2043583Z Ran 1 test in 1.944s 2022-12-01T10:40:43.2043588Z 2022-12-01T10:40:43.2043649Z OK 2022-12-01T10:40:43.2043653Z 2022-12-01T10:40:43.2043739Z Generating XML reports... 2022-12-01T10:40:43.2044116Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104014.xml 2022-12-01T10:40:43.2044388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2044512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2044790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2044929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2045114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplde4b4nc 2022-12-01T10:40:43.2045309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplde4b4nc/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2045567Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2045572Z 2022-12-01T10:40:43.2045643Z Running tests... 2022-12-01T10:40:43.2045825Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2046091Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6742 2022-12-01T10:40:43.2046246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6743 2022-12-01T10:40:43.2046401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6744 2022-12-01T10:40:43.2046550Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6745 2022-12-01T10:40:43.2046828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2046950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2047261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2047399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2047573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4crsoeh 2022-12-01T10:40:43.2047826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4crsoeh/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2048137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2048260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2048541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2048677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2048863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp133enwky 2022-12-01T10:40:43.2049059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp133enwky/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2049319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2049446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2049726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2049865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2050050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35yha18q 2022-12-01T10:40:43.2050247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35yha18q/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2050520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2050644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2050921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2051043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2051227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcelcx3db 2022-12-01T10:40:43.2051427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcelcx3db/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2051592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2051756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2051917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2052078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2052145Z ok (2.243s) 2022-12-01T10:40:43.2052150Z 2022-12-01T10:40:43.2052337Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2052412Z Ran 1 test in 2.243s 2022-12-01T10:40:43.2052417Z 2022-12-01T10:40:43.2052477Z OK 2022-12-01T10:40:43.2052482Z 2022-12-01T10:40:43.2052567Z Generating XML reports... 2022-12-01T10:40:43.2052964Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104018.xml 2022-12-01T10:40:43.2053240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2053363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2053643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2053814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2053987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoy9geh7j 2022-12-01T10:40:43.2054181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoy9geh7j/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2054437Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2054468Z 2022-12-01T10:40:43.2054542Z Running tests... 2022-12-01T10:40:43.2054741Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2054999Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6965 2022-12-01T10:40:43.2055152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6966 2022-12-01T10:40:43.2055307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6967 2022-12-01T10:40:43.2055459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6968 2022-12-01T10:40:43.2055721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2055846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2056127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2056266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2056451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpniqh5s_w 2022-12-01T10:40:43.2056649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpniqh5s_w/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2056921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2057046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2057312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2057448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2057634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprn1eg67v 2022-12-01T10:40:43.2057832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprn1eg67v/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2058103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2058226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2058503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2058640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2058823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88y0cml1 2022-12-01T10:40:43.2059003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88y0cml1/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2059270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2059393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2059678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2059813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2059995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vipqw0z 2022-12-01T10:40:43.2060193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vipqw0z/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2060382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2060534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2060697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2060858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2061082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.2061259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.2061433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.2061731Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.2061903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.2062203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.2062485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.2062774Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.2063041Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.2063305Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.2063576Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.2063840Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.2064098Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.2064355Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.2064424Z ok (2.346s) 2022-12-01T10:40:43.2064429Z 2022-12-01T10:40:43.2064627Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2064705Z Ran 1 test in 2.346s 2022-12-01T10:40:43.2064710Z 2022-12-01T10:40:43.2064758Z OK 2022-12-01T10:40:43.2064776Z 2022-12-01T10:40:43.2064848Z Generating XML reports... 2022-12-01T10:40:43.2065240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104021.xml 2022-12-01T10:40:43.2065516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2065643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2065925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2066062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2066247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtdw58kd 2022-12-01T10:40:43.2066470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtdw58kd/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2066713Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2066731Z 2022-12-01T10:40:43.2066792Z Running tests... 2022-12-01T10:40:43.2066990Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2067252Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7212 2022-12-01T10:40:43.2067435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7213 2022-12-01T10:40:43.2067588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7214 2022-12-01T10:40:43.2067740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7215 2022-12-01T10:40:43.2068016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2068140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2068408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2068543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2068729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9zaw9df 2022-12-01T10:40:43.2068928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9zaw9df/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2069197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2069318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2069596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2069732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2069903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnv3iw1a 2022-12-01T10:40:43.2070097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnv3iw1a/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2070368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2070492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2070768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2070901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2071084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8twz8z21 2022-12-01T10:40:43.2071278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8twz8z21/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2071548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2071658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2071936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2072071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2072251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_t222ea 2022-12-01T10:40:43.2072440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_t222ea/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2072607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2072795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2072959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2073109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2073286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:40:43.2073461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:40:43.2073660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:40:43.2073834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:40:43.2074133Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.2074433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.2074726Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.2075014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:40:43.2075277Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.2075527Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.2075794Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.2076060Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-12-01T10:40:43.2076316Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.2076567Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:40:43.2076638Z ok (2.344s) 2022-12-01T10:40:43.2076644Z 2022-12-01T10:40:43.2076843Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2076917Z Ran 1 test in 2.344s 2022-12-01T10:40:43.2076922Z 2022-12-01T10:40:43.2076982Z OK 2022-12-01T10:40:43.2076987Z 2022-12-01T10:40:43.2077069Z Generating XML reports... 2022-12-01T10:40:43.2077454Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104025.xml 2022-12-01T10:40:43.2077729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2077853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2078132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2078396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2078584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1s4lz5rf 2022-12-01T10:40:43.2078781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1s4lz5rf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2079039Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2079090Z 2022-12-01T10:40:43.2079164Z Running tests... 2022-12-01T10:40:43.2079349Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2079607Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7459 2022-12-01T10:40:43.2079763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7460 2022-12-01T10:40:43.2079953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7461 2022-12-01T10:40:43.2080107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7462 2022-12-01T10:40:43.2080381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2080505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2080790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2080926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2081098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnfjgou5 2022-12-01T10:40:43.2081295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnfjgou5/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2081562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2081688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2081966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2082099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2082284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm089bu1g 2022-12-01T10:40:43.2082479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm089bu1g/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2082734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2082857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2083134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2083271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2083455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpput5ttjb 2022-12-01T10:40:43.2083650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpput5ttjb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2083920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2084045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2084324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2084445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2084627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjl61s3ui 2022-12-01T10:40:43.2084818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjl61s3ui/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2084983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:40:43.2085148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2085308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2085510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2085578Z ok (1.952s) 2022-12-01T10:40:43.2085583Z 2022-12-01T10:40:43.2085769Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2085844Z Ran 1 test in 1.952s 2022-12-01T10:40:43.2085849Z 2022-12-01T10:40:43.2085909Z OK 2022-12-01T10:40:43.2085914Z 2022-12-01T10:40:43.2085997Z Generating XML reports... 2022-12-01T10:40:43.2086390Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104029.xml 2022-12-01T10:40:43.2086696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2086820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2087101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2087238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2087410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_arpxbu 2022-12-01T10:40:43.2087606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_arpxbu/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2087916Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2087921Z 2022-12-01T10:40:43.2087996Z Running tests... 2022-12-01T10:40:43.2088198Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2088486Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7568 2022-12-01T10:40:43.2088644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7569 2022-12-01T10:40:43.2088797Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7570 2022-12-01T10:40:43.2089073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2089183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2089465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2089601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2089789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_t74s56 2022-12-01T10:40:43.2089981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_t74s56/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2090256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2090382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2090664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2090785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2090969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9avr3gpv 2022-12-01T10:40:43.2091240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2091366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2091644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2091777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2091960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygiyt8dl 2022-12-01T10:40:43.2092192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9avr3gpv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2092382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygiyt8dl/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2092533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2092699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2092863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2092959Z ok (2.047s) 2022-12-01T10:40:43.2092964Z 2022-12-01T10:40:43.2093162Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2093238Z Ran 1 test in 2.048s 2022-12-01T10:40:43.2093243Z 2022-12-01T10:40:43.2093303Z OK 2022-12-01T10:40:43.2093308Z 2022-12-01T10:40:43.2093391Z Generating XML reports... 2022-12-01T10:40:43.2093795Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221201104032.xml 2022-12-01T10:40:43.2094069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2094192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2094470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2094605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2094791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbimruwz 2022-12-01T10:40:43.2094986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbimruwz/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2095240Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2095245Z 2022-12-01T10:40:43.2095317Z Running tests... 2022-12-01T10:40:43.2095503Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2095780Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7738 2022-12-01T10:40:43.2095933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7739 2022-12-01T10:40:43.2096086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7740 2022-12-01T10:40:43.2096365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2096488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2096770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2096906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2097093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptiekbzqb 2022-12-01T10:40:43.2097278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptiekbzqb/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2097551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2097674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2097955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2098095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2098280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2kj2ner 2022-12-01T10:40:43.2098475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2kj2ner/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2098784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2098897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2099179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2099316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2099501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrpqvxza 2022-12-01T10:40:43.2099724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrpqvxza/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2099887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2100052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2100213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2100281Z ok (2.047s) 2022-12-01T10:40:43.2100286Z 2022-12-01T10:40:43.2100470Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2100545Z Ran 1 test in 2.047s 2022-12-01T10:40:43.2100550Z 2022-12-01T10:40:43.2100608Z OK 2022-12-01T10:40:43.2100613Z 2022-12-01T10:40:43.2100696Z Generating XML reports... 2022-12-01T10:40:43.2101112Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221201104036.xml 2022-12-01T10:40:43.2101390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2101514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2101795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2101920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2102110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplth5d4yf 2022-12-01T10:40:43.2102304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplth5d4yf/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2102561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-12-01T10:40:43.2102566Z 2022-12-01T10:40:43.2102643Z Running tests... 2022-12-01T10:40:43.2102840Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2103142Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7908 2022-12-01T10:40:43.2103298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7909 2022-12-01T10:40:43.2103453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7910 2022-12-01T10:40:43.2103715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2103838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2104117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2104253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2104440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xyoonyv 2022-12-01T10:40:43.2104632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xyoonyv/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2104904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2105026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2105335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2105458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2105642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfuauzq7 2022-12-01T10:40:43.2105836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfuauzq7/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2106106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:40:43.2106258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:40:43.2106534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:40:43.2106667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:40:43.2106849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4z5e6gar 2022-12-01T10:40:43.2107030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4z5e6gar/_remote_module_non_scriptable.py 2022-12-01T10:40:43.2107193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:40:43.2107356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:40:43.2107515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:40:43.2107790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xyoonyv/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:43.2108062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4z5e6gar/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:43.2108344Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp4z5e6gar/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-12-01T10:40:43.2108415Z ok (2.048s) 2022-12-01T10:40:43.2108420Z 2022-12-01T10:40:43.2108619Z ---------------------------------------------------------------------- 2022-12-01T10:40:43.2108695Z Ran 1 test in 2.049s 2022-12-01T10:40:43.2108700Z 2022-12-01T10:40:43.2108748Z OK 2022-12-01T10:40:43.2108753Z 2022-12-01T10:40:43.2108835Z Generating XML reports... 2022-12-01T10:40:43.2109250Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221201104040.xml 2022-12-01T10:40:43.2109256Z 2022-12-01T10:40:43.2109651Z ##[endgroup] 2022-12-01T10:40:43.2110019Z FINISHED PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_elt7x7yf) 2022-12-01T10:40:43.2110024Z 2022-12-01T10:40:43.2110221Z Running distributed/rpc/test_faulty_agent ... [2022-12-01 10:40:42.683634] 2022-12-01T10:40:43.2110612Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:40:42.683998] 2022-12-01T10:44:52.5046578Z 2022-12-01T10:44:52.5048401Z Expand the folded group to see the log file of distributed/rpc/test_faulty_agent 2022-12-01T10:44:52.5049851Z ##[group]PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent__i9zqla2) 2022-12-01T10:44:52.5050607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2i150l9o 2022-12-01T10:44:52.5051200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2i150l9o/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5052324Z , <__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_verify_backend_options>]> 2022-12-01T10:44:52.5062606Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) 2022-12-01T10:44:52.5063408Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) 2022-12-01T10:44:52.5066948Z , <__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_check_failed_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_messages_to_delay>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_no_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_builtin_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_script_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rref_to_here_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_verify_backend_options>]> 2022-12-01T10:44:52.5069473Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5069814Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5075755Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5076515Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5077048Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5077360Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5077697Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5078118Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5078747Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5079064Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5079455Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5079845Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5105700Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5106170Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5106581Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5107024Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5107505Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5108049Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5108621Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5109100Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) 2022-12-01T10:44:52.5110518Z , <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_script_func>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_to_here_timeout_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_python>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_torchscript_function>]> 2022-12-01T10:44:52.5112007Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-12-01T10:44:52.5112566Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-12-01T10:44:52.5113017Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) 2022-12-01T10:44:52.5113468Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-12-01T10:44:52.5113888Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) 2022-12-01T10:44:52.5114397Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) 2022-12-01T10:44:52.5115296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5115807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5116500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5117089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5117685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6qqh83t 2022-12-01T10:44:52.5118545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6qqh83t/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5119276Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5119577Z 2022-12-01T10:44:52.5119726Z Running tests... 2022-12-01T10:44:52.5120219Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5120876Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8088 2022-12-01T10:44:52.5121573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8089 2022-12-01T10:44:52.5122077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8090 2022-12-01T10:44:52.5122565Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8091 2022-12-01T10:44:52.5123254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5123765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5124434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5124982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5125545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gph2hpy 2022-12-01T10:44:52.5126180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gph2hpy/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5127065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5127634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5128408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5128995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5129618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zts2rp0 2022-12-01T10:44:52.5130499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zts2rp0/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5131345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5131900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5132664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5133396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5133970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4n8agne 2022-12-01T10:44:52.5134625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4n8agne/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5135461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5136017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5136726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5137334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5137948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8x4djasr 2022-12-01T10:44:52.5138654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8x4djasr/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5139261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5139880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5140470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5141042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5141681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:44:52.5142361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:44:52.5143009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-12-01T10:44:52.5143924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:44:52.5144650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-12-01T10:44:52.5145549Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:44:52.5146499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:44:52.5147448Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-12-01T10:44:52.5147956Z ok (9.264s) 2022-12-01T10:44:52.5148125Z 2022-12-01T10:44:52.5148478Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5148872Z Ran 1 test in 9.264s 2022-12-01T10:44:52.5149070Z 2022-12-01T10:44:52.5155669Z OK 2022-12-01T10:44:52.5155822Z 2022-12-01T10:44:52.5155970Z Generating XML reports... 2022-12-01T10:44:52.5156855Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20221201104045.xml 2022-12-01T10:44:52.5157734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5158855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5159564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5160224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5160820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5724xq3k 2022-12-01T10:44:52.5161434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5724xq3k/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5163411Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5163727Z 2022-12-01T10:44:52.5164436Z Running tests... 2022-12-01T10:44:52.5164918Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5165552Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8275 2022-12-01T10:44:52.5166210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8276 2022-12-01T10:44:52.5166794Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8277 2022-12-01T10:44:52.5167288Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8278 2022-12-01T10:44:52.5168011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5168524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5169091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5169437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5169784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_yr0_17 2022-12-01T10:44:52.5170183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_yr0_17/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5170665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5170982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5171407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5171916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5172257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblesa9_v 2022-12-01T10:44:52.5172658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblesa9_v/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5173154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5173479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5173896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5174243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5174583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5avov3w1 2022-12-01T10:44:52.5174981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5avov3w1/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5175455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5175781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5176214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5176546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5176885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhyzqh1t 2022-12-01T10:44:52.5177281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhyzqh1t/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5177724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5178067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5178413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5178764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5179043Z ok (7.854s) 2022-12-01T10:44:52.5179148Z 2022-12-01T10:44:52.5179355Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5179594Z Ran 1 test in 7.854s 2022-12-01T10:44:52.5179706Z 2022-12-01T10:44:52.5179768Z OK 2022-12-01T10:44:52.5179845Z 2022-12-01T10:44:52.5179929Z Generating XML reports... 2022-12-01T10:44:52.5180436Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20221201104055.xml 2022-12-01T10:44:52.5181013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5181333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5181763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5182108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5182456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7dhzidp 2022-12-01T10:44:52.5182846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7dhzidp/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5183308Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5183508Z 2022-12-01T10:44:52.5183580Z Running tests... 2022-12-01T10:44:52.5183870Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5184293Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8450 2022-12-01T10:44:52.5184712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8451 2022-12-01T10:44:52.5185040Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8452 2022-12-01T10:44:52.5185348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8453 2022-12-01T10:44:52.5185795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5186123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5186550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5186881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5187228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8vcrn58 2022-12-01T10:44:52.5187628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8vcrn58/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5188100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5188425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5188851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5189194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5189520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bmh99ey 2022-12-01T10:44:52.5189913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bmh99ey/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5190428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5190758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5191172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5191516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5191856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz3rqi2d 2022-12-01T10:44:52.5192277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz3rqi2d/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5192760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5193083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5193507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5193838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5194179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszuoofqd 2022-12-01T10:44:52.5194576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszuoofqd/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5194940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5195289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5195629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5195970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5196209Z ok (7.855s) 2022-12-01T10:44:52.5196310Z 2022-12-01T10:44:52.5196515Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5196753Z Ran 1 test in 7.855s 2022-12-01T10:44:52.5196865Z 2022-12-01T10:44:52.5196911Z OK 2022-12-01T10:44:52.5197002Z 2022-12-01T10:44:52.5197088Z Generating XML reports... 2022-12-01T10:44:52.5197557Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104105.xml 2022-12-01T10:44:52.5198104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5198612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5199044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5199391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5199723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsgvuibc 2022-12-01T10:44:52.5200131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsgvuibc/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5200601Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5200803Z 2022-12-01T10:44:52.5200876Z Running tests... 2022-12-01T10:44:52.5201172Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5201601Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8625 2022-12-01T10:44:52.5202028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8626 2022-12-01T10:44:52.5202355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8627 2022-12-01T10:44:52.5202663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8628 2022-12-01T10:44:52.5203173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5203510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5203931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5204282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5204628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvrbupew 2022-12-01T10:44:52.5205084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvrbupew/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5205566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5205898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5206420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5206772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5207106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3bwp0fl 2022-12-01T10:44:52.5207595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5207928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5208347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5208694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5209038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2n3r24z 2022-12-01T10:44:52.5209437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3bwp0fl/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5209835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2n3r24z/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5210320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5210646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5211058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5211402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5211741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2zhgj0a 2022-12-01T10:44:52.5212137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2zhgj0a/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5212496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5212847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5213190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5213536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5213772Z ok (7.953s) 2022-12-01T10:44:52.5213874Z 2022-12-01T10:44:52.5214074Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5214311Z Ran 1 test in 7.954s 2022-12-01T10:44:52.5214415Z 2022-12-01T10:44:52.5214475Z OK 2022-12-01T10:44:52.5214568Z 2022-12-01T10:44:52.5214652Z Generating XML reports... 2022-12-01T10:44:52.5215128Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104114.xml 2022-12-01T10:44:52.5215663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5216031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5216455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5216802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5217132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjqwwmmz 2022-12-01T10:44:52.5217534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjqwwmmz/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5218030Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5218237Z 2022-12-01T10:44:52.5218310Z Running tests... 2022-12-01T10:44:52.5218598Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5219006Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8800 2022-12-01T10:44:52.5219414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8801 2022-12-01T10:44:52.5219734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8802 2022-12-01T10:44:52.5220055Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8803 2022-12-01T10:44:52.5220503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5220839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5221263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5221617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5221961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp287brrkh 2022-12-01T10:44:52.5222348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp287brrkh/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5222834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5223161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5223593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5223922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5224268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwy4cj52 2022-12-01T10:44:52.5224661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwy4cj52/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5225142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5225455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5225881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5226225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5226557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_uwszij 2022-12-01T10:44:52.5226952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_uwszij/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5227441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5227765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5228174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5228514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5228885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jp8yh_i 2022-12-01T10:44:52.5229271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jp8yh_i/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5229647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5229995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5230339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5230696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5230946Z ok (12.555s) 2022-12-01T10:44:52.5231049Z 2022-12-01T10:44:52.5231252Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5231483Z Ran 1 test in 12.555s 2022-12-01T10:44:52.5231595Z 2022-12-01T10:44:52.5231658Z OK 2022-12-01T10:44:52.5231748Z 2022-12-01T10:44:52.5231832Z Generating XML reports... 2022-12-01T10:44:52.5232305Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104123.xml 2022-12-01T10:44:52.5232847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5233177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5233603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5233952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5234282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvm_3qm6r 2022-12-01T10:44:52.5234683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvm_3qm6r/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5235144Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5235346Z 2022-12-01T10:44:52.5235406Z Running tests... 2022-12-01T10:44:52.5235706Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5248088Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8975 2022-12-01T10:44:52.5248508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8976 2022-12-01T10:44:52.5248838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8977 2022-12-01T10:44:52.5249175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8978 2022-12-01T10:44:52.5249672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5250009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5250430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5250784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5251133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95h_iu0t 2022-12-01T10:44:52.5251606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5251939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5252369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5252719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5253053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntw0h4k4 2022-12-01T10:44:52.5253452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95h_iu0t/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5253959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntw0h4k4/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5254454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5254769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5255195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5255585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5255915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7q7dpmy 2022-12-01T10:44:52.5256315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7q7dpmy/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5256805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5257136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5257549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5257894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5258236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4thz_eml 2022-12-01T10:44:52.5258619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4thz_eml/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5258997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5259346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5259695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5260035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5260282Z ok (7.853s) 2022-12-01T10:44:52.5260386Z 2022-12-01T10:44:52.5260591Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5260824Z Ran 1 test in 7.854s 2022-12-01T10:44:52.5260937Z 2022-12-01T10:44:52.5260998Z OK 2022-12-01T10:44:52.5261091Z 2022-12-01T10:44:52.5261176Z Generating XML reports... 2022-12-01T10:44:52.5261653Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104137.xml 2022-12-01T10:44:52.5262190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5262523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5262953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5263302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5263634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28grdln4 2022-12-01T10:44:52.5264034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28grdln4/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5264495Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5264697Z 2022-12-01T10:44:52.5264756Z Running tests... 2022-12-01T10:44:52.5265060Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5265466Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9150 2022-12-01T10:44:52.5265875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9151 2022-12-01T10:44:52.5266192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9152 2022-12-01T10:44:52.5266563Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9153 2022-12-01T10:44:52.5267017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5267335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5267762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5268108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5268489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqle_vfwx 2022-12-01T10:44:52.5268874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqle_vfwx/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5269363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5269694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5270122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5270454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5270799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0_idv7w 2022-12-01T10:44:52.5271198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0_idv7w/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5271673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5272001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5272425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5272770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5273101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzbuoay7 2022-12-01T10:44:52.5273502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzbuoay7/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5273985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5274299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5274728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5275073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5275415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vnjug76 2022-12-01T10:44:52.5275799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vnjug76/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5276178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5276528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5276875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5277205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5277451Z ok (2.046s) 2022-12-01T10:44:52.5277555Z 2022-12-01T10:44:52.5277754Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5277981Z Ran 1 test in 2.046s 2022-12-01T10:44:52.5278092Z 2022-12-01T10:44:52.5278154Z OK 2022-12-01T10:44:52.5278403Z 2022-12-01T10:44:52.5278492Z Generating XML reports... 2022-12-01T10:44:52.5278959Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104147.xml 2022-12-01T10:44:52.5279573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5279908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5280334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5280666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5281012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf4ftoqw 2022-12-01T10:44:52.5281454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf4ftoqw/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5281915Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5282103Z 2022-12-01T10:44:52.5282176Z Running tests... 2022-12-01T10:44:52.5282479Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5282880Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9325 2022-12-01T10:44:52.5283266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9326 2022-12-01T10:44:52.5283599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9327 2022-12-01T10:44:52.5283921Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9328 2022-12-01T10:44:52.5284373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5284693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5285120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5285468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5285802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptt6d8hfv 2022-12-01T10:44:52.5286204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptt6d8hfv/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5286754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5287084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5287495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5287843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5288277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ckk1cf_ 2022-12-01T10:44:52.5288679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ckk1cf_/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5289154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5289484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5289911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5290245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5290589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47vy3ibx 2022-12-01T10:44:52.5290992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47vy3ibx/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5291478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5291795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5292220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5292607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5292952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi1qwm2b 2022-12-01T10:44:52.5293337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi1qwm2b/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5293714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5294060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5294425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5294767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5295018Z ok (7.854s) 2022-12-01T10:44:52.5295121Z 2022-12-01T10:44:52.5295322Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5295550Z Ran 1 test in 7.854s 2022-12-01T10:44:52.5295664Z 2022-12-01T10:44:52.5295726Z OK 2022-12-01T10:44:52.5295819Z 2022-12-01T10:44:52.5295904Z Generating XML reports... 2022-12-01T10:44:52.5296361Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104150.xml 2022-12-01T10:44:52.5296911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5297244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5297670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5298004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5298347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp750sdpnq 2022-12-01T10:44:52.5298752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp750sdpnq/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5299196Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5299398Z 2022-12-01T10:44:52.5299472Z Running tests... 2022-12-01T10:44:52.5299778Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5300199Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9500 2022-12-01T10:44:52.5300601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9501 2022-12-01T10:44:52.5300928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9502 2022-12-01T10:44:52.5301255Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9503 2022-12-01T10:44:52.5301692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5302025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5302455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5302802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5303134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh5tqj0c 2022-12-01T10:44:52.5303538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh5tqj0c/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5304026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5304358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5304772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5305116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5305495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tn_a7m6 2022-12-01T10:44:52.5305881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tn_a7m6/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5306367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5306694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5307152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5307483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5307826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtr896xo 2022-12-01T10:44:52.5308224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtr896xo/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5308709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5309022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5309445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5309788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5310119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltgftu56 2022-12-01T10:44:52.5310518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltgftu56/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5310894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5311240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5311578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5311918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5312168Z ok (6.949s) 2022-12-01T10:44:52.5312271Z 2022-12-01T10:44:52.5312458Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5312699Z Ran 1 test in 6.949s 2022-12-01T10:44:52.5312812Z 2022-12-01T10:44:52.5312872Z OK 2022-12-01T10:44:52.5312967Z 2022-12-01T10:44:52.5313050Z Generating XML reports... 2022-12-01T10:44:52.5313506Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104159.xml 2022-12-01T10:44:52.5314050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5314381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5314802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5315150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5315493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyiuy58eh 2022-12-01T10:44:52.5315894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyiuy58eh/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5316342Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5316545Z 2022-12-01T10:44:52.5316618Z Running tests... 2022-12-01T10:44:52.5316919Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5317330Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9675 2022-12-01T10:44:52.5317751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9676 2022-12-01T10:44:52.5318108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9677 2022-12-01T10:44:52.5318613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9678 2022-12-01T10:44:52.5319057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5319389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5319881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5320230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5320561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3z6p4w9d 2022-12-01T10:44:52.5320965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3z6p4w9d/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5321454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5321769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5322201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5322544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5322889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0zxvs21 2022-12-01T10:44:52.5323276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0zxvs21/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5323766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5324095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5324512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5324854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5325200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwp3q5a53 2022-12-01T10:44:52.5325598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwp3q5a53/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5326066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5326454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5326883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5327228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5327558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfac6bdkg 2022-12-01T10:44:52.5327961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfac6bdkg/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5328338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5328677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5329021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5329366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5329937Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-12-01T10:44:52.5330594Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first): 2022-12-01T10:44:52.5331252Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f27297ebceb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5331991Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f27297e76be in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5332829Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7f27297e945e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5333580Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x43a (0x7f272e11e98a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5334558Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5cc (0x7f272e10e82c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5335545Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x12a (0x7f27361bb32a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-12-01T10:44:52.5336469Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x14c (0x7f272e11324c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5337443Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f27361b63c5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-12-01T10:44:52.5338114Z frame #8: + 0x48de4a3 (0x7f272e10c4a3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5338822Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7f272e10d038 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5339638Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f272e107187 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5340226Z frame #11: + 0x490de42 (0x7f272e13be42 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5340725Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f27297da11b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5341095Z frame #13: + 0xdbbf4 (0x7f272965dbf4 in /opt/conda/lib/libstdc++.so.6) 2022-12-01T10:44:52.5341499Z frame #14: + 0x8609 (0x7f273e068609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-12-01T10:44:52.5341870Z frame #15: clone + 0x43 (0x7f273df8d133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-12-01T10:44:52.5342017Z 2022-12-01T10:44:52.5342088Z ok (4.048s) 2022-12-01T10:44:52.5342189Z 2022-12-01T10:44:52.5342417Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5342658Z Ran 1 test in 4.049s 2022-12-01T10:44:52.5342770Z 2022-12-01T10:44:52.5342818Z OK 2022-12-01T10:44:52.5342912Z 2022-12-01T10:44:52.5342998Z Generating XML reports... 2022-12-01T10:44:52.5343474Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104208.xml 2022-12-01T10:44:52.5344028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5344379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5344809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5345156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5345489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvdh3wl9 2022-12-01T10:44:52.5345895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvdh3wl9/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5346359Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5346558Z 2022-12-01T10:44:52.5346632Z Running tests... 2022-12-01T10:44:52.5346918Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5347333Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9850 2022-12-01T10:44:52.5347746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9851 2022-12-01T10:44:52.5348065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9852 2022-12-01T10:44:52.5348395Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9853 2022-12-01T10:44:52.5348843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5349174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5349594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5349940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5350286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnsa4d4c 2022-12-01T10:44:52.5350690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnsa4d4c/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5351160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5351490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5351921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5352252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5352597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp604p3c03 2022-12-01T10:44:52.5352995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp604p3c03/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5353480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5353796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5354228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5354577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5354909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppssrr8fg 2022-12-01T10:44:52.5355351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppssrr8fg/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5355843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5356172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5356583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5356959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5357305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8bcwyp3b 2022-12-01T10:44:52.5357704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8bcwyp3b/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5358069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5359400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5359750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5360082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5360335Z ok (7.862s) 2022-12-01T10:44:52.5360439Z 2022-12-01T10:44:52.5360646Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5360891Z Ran 1 test in 7.862s 2022-12-01T10:44:52.5360991Z 2022-12-01T10:44:52.5361053Z OK 2022-12-01T10:44:52.5361143Z 2022-12-01T10:44:52.5361230Z Generating XML reports... 2022-12-01T10:44:52.5361702Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104213.xml 2022-12-01T10:44:52.5362239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5362571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5363001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5363347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5363679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cp4bvev 2022-12-01T10:44:52.5364083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cp4bvev/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5364544Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5364744Z 2022-12-01T10:44:52.5364805Z Running tests... 2022-12-01T10:44:52.5365109Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5365530Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10025 2022-12-01T10:44:52.5365953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10026 2022-12-01T10:44:52.5366276Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10027 2022-12-01T10:44:52.5366654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10028 2022-12-01T10:44:52.5367108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5367445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5367860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5368206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5368548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvp3zgug 2022-12-01T10:44:52.5369005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvp3zgug/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5369498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5369826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5370253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5370584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5370968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7ke9kmn 2022-12-01T10:44:52.5371449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5371764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5372192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5372539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5372880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpievr9qk7 2022-12-01T10:44:52.5373267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7ke9kmn/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5373676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpievr9qk7/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5374168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5374495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5374905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5375247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5375593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pbyok2a 2022-12-01T10:44:52.5375979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pbyok2a/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5376358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5376700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5377052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5377384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5377632Z ok (7.952s) 2022-12-01T10:44:52.5377736Z 2022-12-01T10:44:52.5377936Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5378164Z Ran 1 test in 7.953s 2022-12-01T10:44:52.5378276Z 2022-12-01T10:44:52.5378338Z OK 2022-12-01T10:44:52.5378430Z 2022-12-01T10:44:52.5378516Z Generating XML reports... 2022-12-01T10:44:52.5378987Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104223.xml 2022-12-01T10:44:52.5379521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5379852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5380279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5380613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5380958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdearm4tt 2022-12-01T10:44:52.5381360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdearm4tt/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5381852Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5382055Z 2022-12-01T10:44:52.5382114Z Running tests... 2022-12-01T10:44:52.5382418Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5382835Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10200 2022-12-01T10:44:52.5383249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10201 2022-12-01T10:44:52.5383596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10202 2022-12-01T10:44:52.5383923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10203 2022-12-01T10:44:52.5384375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5384694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5385123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5385471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5385818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwsw4ujf 2022-12-01T10:44:52.5386205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwsw4ujf/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5386701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5387028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5387444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5387793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5388138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qq0sg09 2022-12-01T10:44:52.5388537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qq0sg09/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5389009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5389334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5389762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5390106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5390434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh2xhotl 2022-12-01T10:44:52.5390833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh2xhotl/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5391318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5391635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5392060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5392406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5392749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ipmof1k 2022-12-01T10:44:52.5393139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ipmof1k/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5393515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5393862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5394196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5394615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5394867Z ok (7.066s) 2022-12-01T10:44:52.5394970Z 2022-12-01T10:44:52.5395173Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5395400Z Ran 1 test in 7.066s 2022-12-01T10:44:52.5395512Z 2022-12-01T10:44:52.5395574Z OK 2022-12-01T10:44:52.5395665Z 2022-12-01T10:44:52.5395751Z Generating XML reports... 2022-12-01T10:44:52.5396214Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104232.xml 2022-12-01T10:44:52.5396795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5397126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5397554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5397889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5398417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1h0g0n9 2022-12-01T10:44:52.5398840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1h0g0n9/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5399312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5399500Z 2022-12-01T10:44:52.5399578Z Running tests... 2022-12-01T10:44:52.5399886Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5400314Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10375 2022-12-01T10:44:52.5400725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10376 2022-12-01T10:44:52.5401058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10377 2022-12-01T10:44:52.5401391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10378 2022-12-01T10:44:52.5401844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5402161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5402590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5402943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5403278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwzrmnjf 2022-12-01T10:44:52.5403681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwzrmnjf/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5404168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5404501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5404922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5405266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5405608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy41n3w1o 2022-12-01T10:44:52.5406012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy41n3w1o/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5406537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5406869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5407300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5407703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5408048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k2rwxzp 2022-12-01T10:44:52.5408452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k2rwxzp/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5408947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5409266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5409737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5410083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5410414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwonbjyjf 2022-12-01T10:44:52.5410814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwonbjyjf/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5411194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5411544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5411880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5412221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5412793Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-12-01T10:44:52.5413404Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first): 2022-12-01T10:44:52.5414058Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7fd095010ceb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5414810Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7fd09500c6be in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5415627Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7fd09500e45e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5416373Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x43a (0x7fd09994398a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5417361Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5cc (0x7fd09993382c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5418352Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x12a (0x7fd0a19e032a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-12-01T10:44:52.5419322Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x14c (0x7fd09993824c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5420295Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7fd0a19db3c5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-12-01T10:44:52.5420953Z frame #8: + 0x48de4a3 (0x7fd0999314a3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5421694Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7fd099932038 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5422503Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7fd09992c187 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5423098Z frame #11: + 0x490de42 (0x7fd099960e42 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5423611Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fd094fff11b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5423969Z frame #13: + 0xdbbf4 (0x7fd094e82bf4 in /opt/conda/lib/libstdc++.so.6) 2022-12-01T10:44:52.5424377Z frame #14: + 0x8609 (0x7fd0a988d609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-12-01T10:44:52.5424749Z frame #15: clone + 0x43 (0x7fd0a97b2133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-12-01T10:44:52.5424909Z 2022-12-01T10:44:52.5424964Z ok (3.958s) 2022-12-01T10:44:52.5425065Z 2022-12-01T10:44:52.5425267Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5425510Z Ran 1 test in 3.958s 2022-12-01T10:44:52.5425624Z 2022-12-01T10:44:52.5425683Z OK 2022-12-01T10:44:52.5425762Z 2022-12-01T10:44:52.5425845Z Generating XML reports... 2022-12-01T10:44:52.5426320Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104241.xml 2022-12-01T10:44:52.5426869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5427189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5427621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5427967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5428310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskayaht1 2022-12-01T10:44:52.5428706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskayaht1/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5429168Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5429368Z 2022-12-01T10:44:52.5429441Z Running tests... 2022-12-01T10:44:52.5429731Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5430132Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10550 2022-12-01T10:44:52.5430535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10551 2022-12-01T10:44:52.5430869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10552 2022-12-01T10:44:52.5431186Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10553 2022-12-01T10:44:52.5431636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5432000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5432434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5432771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5433114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qcd511k 2022-12-01T10:44:52.5433516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qcd511k/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5434022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5434352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5434783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5435134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5435459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmbv0i7r 2022-12-01T10:44:52.5435859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmbv0i7r/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5436347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5436677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5437098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5437446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5437787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bpdug2i 2022-12-01T10:44:52.5438173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bpdug2i/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5438797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5439129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5439560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5439890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5440237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8t671n93 2022-12-01T10:44:52.5440631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8t671n93/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5440991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5441341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5441686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5442030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5442267Z ok (15.472s) 2022-12-01T10:44:52.5442369Z 2022-12-01T10:44:52.5442572Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5442815Z Ran 1 test in 15.472s 2022-12-01T10:44:52.5442927Z 2022-12-01T10:44:52.5442974Z OK 2022-12-01T10:44:52.5443068Z 2022-12-01T10:44:52.5443152Z Generating XML reports... 2022-12-01T10:44:52.5443632Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104246.xml 2022-12-01T10:44:52.5444178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5444492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5444992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5445346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5445677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jmls9qb 2022-12-01T10:44:52.5446080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jmls9qb/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5446601Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5446848Z 2022-12-01T10:44:52.5446923Z Running tests... 2022-12-01T10:44:52.5447213Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5447613Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10725 2022-12-01T10:44:52.5448012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10726 2022-12-01T10:44:52.5448346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10727 2022-12-01T10:44:52.5448662Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10728 2022-12-01T10:44:52.5449114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5449443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5449857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5450204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5450550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2kfkawpr 2022-12-01T10:44:52.5450954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2kfkawpr/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5451432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5451761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5452188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5452521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5452863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpob7t1xbs 2022-12-01T10:44:52.5453265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpob7t1xbs/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5453750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5454067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5454501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5454853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5455193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1rm_518 2022-12-01T10:44:52.5455576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1rm_518/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5456060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5456393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5456803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5457146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5457487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiyruxqa 2022-12-01T10:44:52.5457915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiyruxqa/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5458283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5458630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5458978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5459355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5459593Z ok (9.459s) 2022-12-01T10:44:52.5459696Z 2022-12-01T10:44:52.5459898Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5460137Z Ran 1 test in 9.459s 2022-12-01T10:44:52.5460236Z 2022-12-01T10:44:52.5460298Z OK 2022-12-01T10:44:52.5460389Z 2022-12-01T10:44:52.5460475Z Generating XML reports... 2022-12-01T10:44:52.5460952Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104303.xml 2022-12-01T10:44:52.5461485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5461814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5462241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5462590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5462923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij_vvm7f 2022-12-01T10:44:52.5463324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij_vvm7f/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5463783Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5463982Z 2022-12-01T10:44:52.5464056Z Running tests... 2022-12-01T10:44:52.5464350Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5464751Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10900 2022-12-01T10:44:52.5465149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10901 2022-12-01T10:44:52.5465466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10902 2022-12-01T10:44:52.5465798Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10903 2022-12-01T10:44:52.5466253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5466583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5466996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5467344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5467692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcz51z2cq 2022-12-01T10:44:52.5468081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcz51z2cq/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5468567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5468900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5469328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5469662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5470006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdr8gwnp 2022-12-01T10:44:52.5470515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5470846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5471260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5471609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5471954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocnkhbxc 2022-12-01T10:44:52.5472377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdr8gwnp/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5472783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocnkhbxc/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5473271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5473603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5474025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5474375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5474718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8d_k1lwa 2022-12-01T10:44:52.5475101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8d_k1lwa/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5475480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5475830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5476176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5476510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5476757Z ok (3.947s) 2022-12-01T10:44:52.5476859Z 2022-12-01T10:44:52.5477063Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5477291Z Ran 1 test in 3.947s 2022-12-01T10:44:52.5477403Z 2022-12-01T10:44:52.5477466Z OK 2022-12-01T10:44:52.5477556Z 2022-12-01T10:44:52.5477639Z Generating XML reports... 2022-12-01T10:44:52.5478117Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104314.xml 2022-12-01T10:44:52.5478853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5479186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5479614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5479963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5480298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgjx3i_l 2022-12-01T10:44:52.5480702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgjx3i_l/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5481166Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5481367Z 2022-12-01T10:44:52.5481427Z Running tests... 2022-12-01T10:44:52.5481730Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5482152Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11075 2022-12-01T10:44:52.5482567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11076 2022-12-01T10:44:52.5482888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11077 2022-12-01T10:44:52.5483218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11078 2022-12-01T10:44:52.5483744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5484066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5484500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5484845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5485226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3x9cnn47 2022-12-01T10:44:52.5485611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3x9cnn47/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5486101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5486485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5486925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5487256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5487598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qazj7x7 2022-12-01T10:44:52.5487998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qazj7x7/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5488469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5488801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5489235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5489580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5489917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5gg9dop 2022-12-01T10:44:52.5490327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5gg9dop/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5491166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5491497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5491915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5492263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5492606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5w7nqil 2022-12-01T10:44:52.5492995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5w7nqil/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5493367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5493721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5494064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5494395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5494637Z ok (7.953s) 2022-12-01T10:44:52.5494739Z 2022-12-01T10:44:52.5494949Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5495179Z Ran 1 test in 7.953s 2022-12-01T10:44:52.5495292Z 2022-12-01T10:44:52.5495353Z OK 2022-12-01T10:44:52.5495443Z 2022-12-01T10:44:52.5495528Z Generating XML reports... 2022-12-01T10:44:52.5496008Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104319.xml 2022-12-01T10:44:52.5496542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5496926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5497357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5497687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5498030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnxiukv5 2022-12-01T10:44:52.5498470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnxiukv5/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5498931Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5499131Z 2022-12-01T10:44:52.5499191Z Running tests... 2022-12-01T10:44:52.5499495Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5499919Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11250 2022-12-01T10:44:52.5500337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11251 2022-12-01T10:44:52.5500658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11252 2022-12-01T10:44:52.5500982Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11253 2022-12-01T10:44:52.5501438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5501757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5502184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5502537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5502884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jhavtrx 2022-12-01T10:44:52.5503272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jhavtrx/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5503762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5504092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5504507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5504853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5505196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pcsabwp 2022-12-01T10:44:52.5505596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pcsabwp/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5506069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5506397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5506826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5507169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5507493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1y4i05rr 2022-12-01T10:44:52.5507892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1y4i05rr/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5508376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5508690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5509116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5509495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5509842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwx_hrbr7 2022-12-01T10:44:52.5510225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwx_hrbr7/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5510600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5510947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5511305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5511646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5512218Z [E request_callback_no_python.cpp:559] Received error while processing request type 261: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-12-01T10:44:52.5512827Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first): 2022-12-01T10:44:52.5513487Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f99b1939ceb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5514220Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f99b19356be in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5515028Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7f99b193745e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5515776Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x43a (0x7f99b626c98a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5516754Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5cc (0x7f99b625c82c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5517736Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0xcc (0x7f99be304cbc in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-12-01T10:44:52.5518919Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7f99b6261294 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5519905Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f99be3043c5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-12-01T10:44:52.5520564Z frame #8: + 0x48de4a3 (0x7f99b625a4a3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5521344Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7f99b625b038 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5522156Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f99b6255187 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5522748Z frame #11: + 0x490de42 (0x7f99b6289e42 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-12-01T10:44:52.5523303Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f99b192811b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-12-01T10:44:52.5523661Z frame #13: + 0xdbbf4 (0x7f99b17abbf4 in /opt/conda/lib/libstdc++.so.6) 2022-12-01T10:44:52.5524070Z frame #14: + 0x8609 (0x7f99c61b6609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-12-01T10:44:52.5524442Z frame #15: clone + 0x43 (0x7f99c60db133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-12-01T10:44:52.5524602Z 2022-12-01T10:44:52.5524888Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #0 to worker0, as the agent is shutting down 2022-12-01T10:44:52.5525151Z ok (4.948s) 2022-12-01T10:44:52.5525253Z 2022-12-01T10:44:52.5525454Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5525695Z Ran 1 test in 4.948s 2022-12-01T10:44:52.5525806Z 2022-12-01T10:44:52.5525856Z OK 2022-12-01T10:44:52.5525946Z 2022-12-01T10:44:52.5526031Z Generating XML reports... 2022-12-01T10:44:52.5526576Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104329.xml 2022-12-01T10:44:52.5527132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5527453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5527886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5528236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5528569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6afdq1_q 2022-12-01T10:44:52.5528972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6afdq1_q/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5529438Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5529641Z 2022-12-01T10:44:52.5529714Z Running tests... 2022-12-01T10:44:52.5530007Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5530425Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11425 2022-12-01T10:44:52.5530839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11426 2022-12-01T10:44:52.5531170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11427 2022-12-01T10:44:52.5531484Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11428 2022-12-01T10:44:52.5531932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5532261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5532681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5533028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5533372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54k16dkk 2022-12-01T10:44:52.5533775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54k16dkk/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5534308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5534636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5535062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5535394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5535768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprt9g8yun 2022-12-01T10:44:52.5536167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprt9g8yun/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5536656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5536971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5537397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5537739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5538082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1g1mvyue 2022-12-01T10:44:52.5538471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1g1mvyue/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5538959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5539292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5539705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5539840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5540027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoimf97_f 2022-12-01T10:44:52.5540222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoimf97_f/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5540387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5540552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5540711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5540875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5540929Z ok (7.956s) 2022-12-01T10:44:52.5540948Z 2022-12-01T10:44:52.5541137Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5541212Z Ran 1 test in 7.957s 2022-12-01T10:44:52.5541217Z 2022-12-01T10:44:52.5541278Z OK 2022-12-01T10:44:52.5541283Z 2022-12-01T10:44:52.5541367Z Generating XML reports... 2022-12-01T10:44:52.5541720Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104335.xml 2022-12-01T10:44:52.5541995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5542119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5542398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5542523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5542707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvb19febv 2022-12-01T10:44:52.5542902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvb19febv/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5543148Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5543153Z 2022-12-01T10:44:52.5543257Z Running tests... 2022-12-01T10:44:52.5543459Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5543718Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11600 2022-12-01T10:44:52.5543874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11601 2022-12-01T10:44:52.5544060Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11602 2022-12-01T10:44:52.5544202Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11603 2022-12-01T10:44:52.5544476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5544599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5544881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5545015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5545199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ejoniw4 2022-12-01T10:44:52.5545393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ejoniw4/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5545669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5545784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5546064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5546203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5546389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflc3p5mg 2022-12-01T10:44:52.5546588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflc3p5mg/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5546860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5546984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5547263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5547401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5547571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7id477u 2022-12-01T10:44:52.5547768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7id477u/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5548037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5548163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5548443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5548577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5548760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ufro9t5 2022-12-01T10:44:52.5548953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ufro9t5/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5549107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5549271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5549432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5549591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5549688Z ok (7.854s) 2022-12-01T10:44:52.5549694Z 2022-12-01T10:44:52.5549894Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5549972Z Ran 1 test in 7.854s 2022-12-01T10:44:52.5549977Z 2022-12-01T10:44:52.5550040Z OK 2022-12-01T10:44:52.5550044Z 2022-12-01T10:44:52.5550127Z Generating XML reports... 2022-12-01T10:44:52.5550469Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104345.xml 2022-12-01T10:44:52.5550771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5550895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5551174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5551311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5551497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryydki2z 2022-12-01T10:44:52.5551694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryydki2z/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5551941Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5551946Z 2022-12-01T10:44:52.5552006Z Running tests... 2022-12-01T10:44:52.5552206Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5552448Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11775 2022-12-01T10:44:52.5552606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11776 2022-12-01T10:44:52.5552760Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11777 2022-12-01T10:44:52.5552917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11778 2022-12-01T10:44:52.5553189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5553311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5553591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5553715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5553901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4j_5izl 2022-12-01T10:44:52.5554096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4j_5izl/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5554371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5554495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5554777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5554911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5555094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcq2do6bz 2022-12-01T10:44:52.5555275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcq2do6bz/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5555552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5555675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5555961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5556096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5556313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8o61oyc 2022-12-01T10:44:52.5556504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8o61oyc/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5556778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5556899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5557192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5557328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5557513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48r8ochi 2022-12-01T10:44:52.5557704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48r8ochi/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5557871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5558032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5558309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5558539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5558596Z ok (7.959s) 2022-12-01T10:44:52.5558615Z 2022-12-01T10:44:52.5558808Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5558886Z Ran 1 test in 7.959s 2022-12-01T10:44:52.5558892Z 2022-12-01T10:44:52.5558952Z OK 2022-12-01T10:44:52.5558957Z 2022-12-01T10:44:52.5559044Z Generating XML reports... 2022-12-01T10:44:52.5559400Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104354.xml 2022-12-01T10:44:52.5559678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5559804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5560085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5560208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5560392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcqz39g8 2022-12-01T10:44:52.5560588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcqz39g8/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5560834Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5560840Z 2022-12-01T10:44:52.5560913Z Running tests... 2022-12-01T10:44:52.5561111Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5561359Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11950 2022-12-01T10:44:52.5561516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11951 2022-12-01T10:44:52.5561671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11952 2022-12-01T10:44:52.5561811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11953 2022-12-01T10:44:52.5562083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5562207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5562492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5562626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5562874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijj1j4jt 2022-12-01T10:44:52.5563071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijj1j4jt/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5563346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5563456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5563739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5563914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5564101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_gu69zg 2022-12-01T10:44:52.5564296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_gu69zg/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5564570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5564695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5564975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5565108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5565279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpho_6n1jz 2022-12-01T10:44:52.5565475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpho_6n1jz/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5565748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5565870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5566149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5566285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5566523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzmqns2r 2022-12-01T10:44:52.5566719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzmqns2r/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5566871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5567038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5567199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5567361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5567430Z ok (7.853s) 2022-12-01T10:44:52.5567434Z 2022-12-01T10:44:52.5567637Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5567713Z Ran 1 test in 7.853s 2022-12-01T10:44:52.5567720Z 2022-12-01T10:44:52.5567781Z OK 2022-12-01T10:44:52.5567786Z 2022-12-01T10:44:52.5567870Z Generating XML reports... 2022-12-01T10:44:52.5568221Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104403.xml 2022-12-01T10:44:52.5568493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5568619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5568896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5569030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5569215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwcd65pw 2022-12-01T10:44:52.5569453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwcd65pw/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5569703Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5569708Z 2022-12-01T10:44:52.5569780Z Running tests... 2022-12-01T10:44:52.5569964Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5570208Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12125 2022-12-01T10:44:52.5570397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12126 2022-12-01T10:44:52.5570552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12127 2022-12-01T10:44:52.5570706Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12128 2022-12-01T10:44:52.5570979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5571107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5571386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5571510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5571693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcjso1at 2022-12-01T10:44:52.5571888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcjso1at/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5572163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5572285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5572563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5572699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5572883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb92pekbg 2022-12-01T10:44:52.5573078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb92pekbg/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5573340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5573463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5573744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5573878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5574061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4vkfc20 2022-12-01T10:44:52.5574256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4vkfc20/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5574526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5574649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5574915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5575049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5575238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogyuz36v 2022-12-01T10:44:52.5575436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogyuz36v/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5575599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5575759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5575966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5576131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5576198Z ok (7.958s) 2022-12-01T10:44:52.5576203Z 2022-12-01T10:44:52.5576388Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5576462Z Ran 1 test in 7.959s 2022-12-01T10:44:52.5576467Z 2022-12-01T10:44:52.5576526Z OK 2022-12-01T10:44:52.5576557Z 2022-12-01T10:44:52.5576641Z Generating XML reports... 2022-12-01T10:44:52.5577006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104413.xml 2022-12-01T10:44:52.5577280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5577402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5577684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5577807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5577993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7uxl7nc 2022-12-01T10:44:52.5578189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7uxl7nc/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5578433Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5578440Z 2022-12-01T10:44:52.5578514Z Running tests... 2022-12-01T10:44:52.5578714Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5578966Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12300 2022-12-01T10:44:52.5579124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12301 2022-12-01T10:44:52.5579281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12302 2022-12-01T10:44:52.5579422Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12303 2022-12-01T10:44:52.5579698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5579822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5580107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5580242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5580427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70la5k2e 2022-12-01T10:44:52.5580625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70la5k2e/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5580901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5581011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5581289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5581423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5581608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy8ybkok 2022-12-01T10:44:52.5581804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy8ybkok/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5582073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5582194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5582518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5582655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5582826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mb90i2l 2022-12-01T10:44:52.5583019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mb90i2l/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5583285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5583438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5583720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5583854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5584039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwevvgtf7 2022-12-01T10:44:52.5584235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwevvgtf7/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5584386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5584550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5584713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5584879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5584951Z ok (7.954s) 2022-12-01T10:44:52.5584956Z 2022-12-01T10:44:52.5585156Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5585230Z Ran 1 test in 7.955s 2022-12-01T10:44:52.5585235Z 2022-12-01T10:44:52.5585295Z OK 2022-12-01T10:44:52.5585300Z 2022-12-01T10:44:52.5585382Z Generating XML reports... 2022-12-01T10:44:52.5585733Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104422.xml 2022-12-01T10:44:52.5586008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5586132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5586417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5586557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5586741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d0ftkxj 2022-12-01T10:44:52.5586935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d0ftkxj/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5587186Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5587191Z 2022-12-01T10:44:52.5587263Z Running tests... 2022-12-01T10:44:52.5587451Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5587698Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12475 2022-12-01T10:44:52.5587855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12476 2022-12-01T10:44:52.5588009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12477 2022-12-01T10:44:52.5588163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12478 2022-12-01T10:44:52.5588434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5588558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5588838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5588992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5589177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p9it78k 2022-12-01T10:44:52.5589373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p9it78k/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5589649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5589813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5590092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5590226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5590413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpravutj3_ 2022-12-01T10:44:52.5590609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpravutj3_/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5590869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5590989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5591268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5591405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5591589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3a_wmvq 2022-12-01T10:44:52.5591783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3a_wmvq/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5592052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5592174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5592442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5592577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5592760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpju_5idms 2022-12-01T10:44:52.5592953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpju_5idms/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5593121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5593279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5593442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5593603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5593669Z ok (4.047s) 2022-12-01T10:44:52.5593677Z 2022-12-01T10:44:52.5593860Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5593935Z Ran 1 test in 4.047s 2022-12-01T10:44:52.5593940Z 2022-12-01T10:44:52.5593999Z OK 2022-12-01T10:44:52.5594003Z 2022-12-01T10:44:52.5594088Z Generating XML reports... 2022-12-01T10:44:52.5594457Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104432.xml 2022-12-01T10:44:52.5594732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5594857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5595133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5595255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5595470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8u3f9sjn 2022-12-01T10:44:52.5595666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8u3f9sjn/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5595913Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5595919Z 2022-12-01T10:44:52.5595991Z Running tests... 2022-12-01T10:44:52.5596188Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5596457Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12650 2022-12-01T10:44:52.5596616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12651 2022-12-01T10:44:52.5596771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12652 2022-12-01T10:44:52.5596911Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12653 2022-12-01T10:44:52.5597190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5597312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5597591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5597727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5597916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_56mrx09 2022-12-01T10:44:52.5598108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_56mrx09/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5598520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5598632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5598916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5599052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5599237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppirin3k4 2022-12-01T10:44:52.5599432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppirin3k4/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5599708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5599832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5600109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5600244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5600422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppscu7i73 2022-12-01T10:44:52.5600617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppscu7i73/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5600889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5601013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5601295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5601434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5601622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6498lrj7 2022-12-01T10:44:52.5601814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6498lrj7/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5601977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5602187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5602352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5602512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5602581Z ok (6.451s) 2022-12-01T10:44:52.5602585Z 2022-12-01T10:44:52.5602786Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5602897Z Ran 1 test in 6.451s 2022-12-01T10:44:52.5602903Z 2022-12-01T10:44:52.5602964Z OK 2022-12-01T10:44:52.5602968Z 2022-12-01T10:44:52.5603052Z Generating XML reports... 2022-12-01T10:44:52.5603406Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104437.xml 2022-12-01T10:44:52.5603683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5603808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5604088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5604223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5604409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffnpp7er 2022-12-01T10:44:52.5604607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffnpp7er/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5604852Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-12-01T10:44:52.5604858Z 2022-12-01T10:44:52.5604930Z Running tests... 2022-12-01T10:44:52.5605113Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5605365Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12825 2022-12-01T10:44:52.5605522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12826 2022-12-01T10:44:52.5605676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12827 2022-12-01T10:44:52.5605829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12828 2022-12-01T10:44:52.5606101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5606227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5606567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5606690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5606874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tzi6_v7 2022-12-01T10:44:52.5607069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tzi6_v7/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5607344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5607467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5607748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5607888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5608075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcumtkys 2022-12-01T10:44:52.5608270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcumtkys/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5608528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5608687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5608971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5609104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5609287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94c8qg8b 2022-12-01T10:44:52.5609477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94c8qg8b/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5609775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:44:52.5609895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:44:52.5610158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:44:52.5610296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:44:52.5610479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitng0qw4 2022-12-01T10:44:52.5610672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitng0qw4/_remote_module_non_scriptable.py 2022-12-01T10:44:52.5610835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:44:52.5610993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-12-01T10:44:52.5611156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:44:52.5611317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-12-01T10:44:52.5611384Z ok (6.452s) 2022-12-01T10:44:52.5611390Z 2022-12-01T10:44:52.5611576Z ---------------------------------------------------------------------- 2022-12-01T10:44:52.5611650Z Ran 1 test in 6.452s 2022-12-01T10:44:52.5611655Z 2022-12-01T10:44:52.5611716Z OK 2022-12-01T10:44:52.5611721Z 2022-12-01T10:44:52.5611807Z Generating XML reports... 2022-12-01T10:44:52.5612173Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104445.xml 2022-12-01T10:44:52.5612178Z 2022-12-01T10:44:52.5612514Z ##[endgroup] 2022-12-01T10:44:52.5612867Z FINISHED PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent__i9zqla2) 2022-12-01T10:44:52.5612874Z 2022-12-01T10:44:52.5613062Z Running distributed/test_c10d_pypg ... [2022-12-01 10:44:52.507242] 2022-12-01T10:44:52.5613400Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:44:52.507685] 2022-12-01T10:45:49.2877062Z 2022-12-01T10:45:49.2877607Z Expand the folded group to see the log file of distributed/test_c10d_pypg 2022-12-01T10:45:49.2948604Z ##[group]PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_8nfmbgm7) 2022-12-01T10:45:49.2949295Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg 2022-12-01T10:45:49.2949617Z 2022-12-01T10:45:49.2949753Z Running tests... 2022-12-01T10:45:49.2950120Z ---------------------------------------------------------------------- 2022-12-01T10:45:49.2950435Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2951286Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13004 2022-12-01T10:45:49.2952158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2952663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2953522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2954060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2954393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2954684Z skip: Need at least 2 CUDA devices (1.646s) 2022-12-01T10:45:49.2954981Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2955463Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13015 2022-12-01T10:45:49.2955995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2956328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2956764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2957100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2957425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2957713Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.2957997Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2958590Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13026 2022-12-01T10:45:49.2959106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2959436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2959849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2960198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2960517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2960791Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.2961081Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2961494Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13037 2022-12-01T10:45:49.2962007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2962323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2962750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2963093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2963403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2963689Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.2963999Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2964530Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13048 2022-12-01T10:45:49.2965036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2965363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2965789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2966192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2966561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2966853Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.2967164Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2967686Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13059 2022-12-01T10:45:49.2968246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2968572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2968994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2969325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2969649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2969934Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.2970216Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2970749Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13070 2022-12-01T10:45:49.2971274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2971606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2972022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2972365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2972685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2972976Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.2973258Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2973795Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13081 2022-12-01T10:45:49.2974325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2974644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2975075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2975418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2975738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2976014Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.2976303Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2976723Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13092 2022-12-01T10:45:49.2977243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2977562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2977986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2978325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2978634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2978920Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.2979256Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2979690Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13103 2022-12-01T10:45:49.2980204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2980533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2980991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2981462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2981772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2982062Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.2982368Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2982785Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13114 2022-12-01T10:45:49.2983307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2983634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2984065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2984399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2984718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2985004Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.2985299Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2985711Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13125 2022-12-01T10:45:49.2986211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2986538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2986951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2987300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2987622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2987908Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.2988199Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-12-01T10:45:49.2988614Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13136 2022-12-01T10:45:49.2989116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2989432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2989861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2990209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2990532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2990805Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.2991168Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13147 2022-12-01T10:45:49.2991716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2992043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2992507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2992853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2993206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2993567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54x01xs_ 2022-12-01T10:45:49.2993968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54x01xs_/_remote_module_non_scriptable.py 2022-12-01T10:45:49.2994246Z ok (1.508s) 2022-12-01T10:45:49.2994569Z test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13158 2022-12-01T10:45:49.2995067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2995400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2995825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.2996157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.2996481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.2996850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7c172zm 2022-12-01T10:45:49.2997246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7c172zm/_remote_module_non_scriptable.py 2022-12-01T10:45:49.2997612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:45:49.2997872Z ok (1.508s) 2022-12-01T10:45:49.2998418Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13169 2022-12-01T10:45:49.2998935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.2999264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.2999693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3000038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3000346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3000716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4yv1sku 2022-12-01T10:45:49.3001121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4yv1sku/_remote_module_non_scriptable.py 2022-12-01T10:45:49.3001502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:45:49.3001747Z ok (1.508s) 2022-12-01T10:45:49.3002075Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13180 2022-12-01T10:45:49.3002597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3002913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3003338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3003680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3004000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3004641Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3005471Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3006381Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3007190Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3007997Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3008801Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3009272Z ok (1.408s) 2022-12-01T10:45:49.3009604Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13191 2022-12-01T10:45:49.3010118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3010452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3010879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3011208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3011529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3011820Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3012194Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13202 2022-12-01T10:45:49.3012705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3013031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3013454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3013798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3014105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3014394Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3014676Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3015222Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13213 2022-12-01T10:45:49.3015746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3016071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3016498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3016857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3017179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3017469Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3017747Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3018169Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13224 2022-12-01T10:45:49.3018685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3019012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3019422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3019771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3020091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3020365Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3020658Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3021071Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13235 2022-12-01T10:45:49.3021582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3021896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3022325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3022669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3022993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3023265Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3023555Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3023960Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13246 2022-12-01T10:45:49.3024456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3024782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3025207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3025552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3025860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3026148Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3026456Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3026970Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13257 2022-12-01T10:45:49.3027532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3027861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3028286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3028619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3028941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3029262Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3029568Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3030088Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13268 2022-12-01T10:45:49.3030606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3030934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3031346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3031691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3032010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3032301Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.3032584Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3033117Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13279 2022-12-01T10:45:49.3033643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3033974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3034386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3034731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3035052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3035328Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.3035624Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3036160Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13290 2022-12-01T10:45:49.3036689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3037006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3037432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3037775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3038083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3038499Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3038790Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3039208Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13301 2022-12-01T10:45:49.3039719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3040048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3040532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3040875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3041185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3041473Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3041777Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3042229Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13312 2022-12-01T10:45:49.3042750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3043081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3043511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3043842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3073840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3074259Z skip: Need at least 2 CUDA devices (1.407s) 2022-12-01T10:45:49.3074552Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3075008Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13323 2022-12-01T10:45:49.3075601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3075926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3076367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3076719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3077044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3077324Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3077633Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3078046Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13334 2022-12-01T10:45:49.3078671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3079006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3079432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3079780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3080091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3080378Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3080677Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-12-01T10:45:49.3081089Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13345 2022-12-01T10:45:49.3081580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3081906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3082331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3082780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3083103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3083392Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3083757Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13356 2022-12-01T10:45:49.3084267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3084645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3085077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3085411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3085729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3086162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnb2i5zrx 2022-12-01T10:45:49.3086571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnb2i5zrx/_remote_module_non_scriptable.py 2022-12-01T10:45:49.3086839Z ok (1.408s) 2022-12-01T10:45:49.3087163Z test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13367 2022-12-01T10:45:49.3087682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3088010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3088428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3088774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3089100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3089462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zcz7lkp 2022-12-01T10:45:49.3089862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zcz7lkp/_remote_module_non_scriptable.py 2022-12-01T10:45:49.3090248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:45:49.3090505Z ok (1.508s) 2022-12-01T10:45:49.3090827Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13378 2022-12-01T10:45:49.3091354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3091681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3092098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3092445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3092766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3093136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplol4qe20 2022-12-01T10:45:49.3093527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplol4qe20/_remote_module_non_scriptable.py 2022-12-01T10:45:49.3093914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:45:49.3094168Z ok (1.508s) 2022-12-01T10:45:49.3094484Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13389 2022-12-01T10:45:49.3095000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3095328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3095790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3096124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3096445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3097043Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3097902Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3098715Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3099522Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3100305Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3101107Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-12-01T10:45:49.3101578Z ok (1.508s) 2022-12-01T10:45:49.3101910Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13400 2022-12-01T10:45:49.3102434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3102750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3103179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3103524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3103849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3104126Z skip: Need at least 2 CUDA devices (1.408s) 2022-12-01T10:45:49.3104495Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13411 2022-12-01T10:45:49.3105019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:45:49.3105334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:45:49.3105762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:45:49.3106137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:45:49.3106461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:45:49.3106738Z skip: Need at least 2 CUDA devices (1.508s) 2022-12-01T10:45:49.3106874Z 2022-12-01T10:45:49.3107077Z ---------------------------------------------------------------------- 2022-12-01T10:45:49.3107319Z Ran 38 tests in 55.245s 2022-12-01T10:45:49.3107434Z 2022-12-01T10:45:49.3107524Z OK (skipped=30) 2022-12-01T10:45:49.3107631Z 2022-12-01T10:45:49.3107717Z Generating XML reports... 2022-12-01T10:45:49.3108172Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20221201104453.xml 2022-12-01T10:45:49.3108766Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20221201104453.xml 2022-12-01T10:45:49.3109027Z 2022-12-01T10:45:49.3109321Z ##[endgroup] 2022-12-01T10:45:49.3109748Z FINISHED PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_8nfmbgm7) 2022-12-01T10:45:49.3109991Z 2022-12-01T10:45:49.3110183Z Running distributed/test_c10d_spawn_gloo ... [2022-12-01 10:45:49.288378] 2022-12-01T10:45:49.3110708Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:45:49.288724] 2022-12-01T10:46:23.5861237Z 2022-12-01T10:46:23.5862203Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo 2022-12-01T10:46:23.5863237Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_r5vhj01r) 2022-12-01T10:46:23.5863907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0ad2wxa 2022-12-01T10:46:23.5864483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0ad2wxa/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5865185Z , <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]> 2022-12-01T10:46:23.5865854Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) 2022-12-01T10:46:23.5866417Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) 2022-12-01T10:46:23.5866768Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) 2022-12-01T10:46:23.5867485Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]> 2022-12-01T10:46:23.5868168Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) 2022-12-01T10:46:23.5868541Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) 2022-12-01T10:46:23.5868855Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) 2022-12-01T10:46:23.5869216Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) 2022-12-01T10:46:23.5869508Z 2022-12-01T10:46:23.5869867Z 2022-12-01T10:46:23.5870998Z , <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]> 2022-12-01T10:46:23.5871883Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) 2022-12-01T10:46:23.5872179Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) 2022-12-01T10:46:23.5872465Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) 2022-12-01T10:46:23.5872799Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) 2022-12-01T10:46:23.5873096Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) 2022-12-01T10:46:23.5873521Z test_gather (__main__.TestDistributedNNFunctionsGloo) 2022-12-01T10:46:23.5873808Z test_reduce (__main__.TestDistributedNNFunctionsGloo) 2022-12-01T10:46:23.5874094Z test_scatter (__main__.TestDistributedNNFunctionsGloo) 2022-12-01T10:46:23.5874639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5874963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5875402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5875753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5876083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_tw2lvv 2022-12-01T10:46:23.5876487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_tw2lvv/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5876944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5877144Z 2022-12-01T10:46:23.5877217Z Running tests... 2022-12-01T10:46:23.5877509Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5877949Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-12-01T10:46:23.5878529Z ok (0.012s) 2022-12-01T10:46:23.5878633Z 2022-12-01T10:46:23.5878825Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5879062Z Ran 1 test in 0.012s 2022-12-01T10:46:23.5879183Z 2022-12-01T10:46:23.5879279Z OK 2022-12-01T10:46:23.5879374Z 2022-12-01T10:46:23.5879459Z Generating XML reports... 2022-12-01T10:46:23.5880032Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221201104551.xml 2022-12-01T10:46:23.5880635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5881026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5881459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5881837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5882188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r83qeiq 2022-12-01T10:46:23.5882637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r83qeiq/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5883130Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5883328Z 2022-12-01T10:46:23.5883401Z Running tests... 2022-12-01T10:46:23.5883705Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5884129Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.000s) 2022-12-01T10:46:23.5884345Z 2022-12-01T10:46:23.5884589Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5884828Z Ran 1 test in 0.001s 2022-12-01T10:46:23.5884938Z 2022-12-01T10:46:23.5885011Z OK (skipped=1) 2022-12-01T10:46:23.5885119Z 2022-12-01T10:46:23.5885189Z Generating XML reports... 2022-12-01T10:46:23.5885848Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221201104552.xml 2022-12-01T10:46:23.5886561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5886943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5887361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5887818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5888161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6e70cb2 2022-12-01T10:46:23.5888602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6e70cb2/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5889072Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5889305Z 2022-12-01T10:46:23.5889381Z Running tests... 2022-12-01T10:46:23.5889685Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5890203Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.001s) 2022-12-01T10:46:23.5890575Z 2022-12-01T10:46:23.5890898Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5891273Z Ran 1 test in 0.001s 2022-12-01T10:46:23.5891442Z 2022-12-01T10:46:23.5891541Z OK (skipped=1) 2022-12-01T10:46:23.5891697Z 2022-12-01T10:46:23.5891833Z Generating XML reports... 2022-12-01T10:46:23.5892611Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221201104554.xml 2022-12-01T10:46:23.5893539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5894100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5894808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5895282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5895764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhwh6z5i 2022-12-01T10:46:23.5896330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhwh6z5i/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5897063Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5897401Z 2022-12-01T10:46:23.5897513Z Running tests... 2022-12-01T10:46:23.5897982Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5898522Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-12-01T10:46:23.5898860Z 2022-12-01T10:46:23.5899168Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5899516Z Ran 1 test in 0.000s 2022-12-01T10:46:23.5899663Z 2022-12-01T10:46:23.5899763Z OK (skipped=1) 2022-12-01T10:46:23.5899930Z 2022-12-01T10:46:23.5900070Z Generating XML reports... 2022-12-01T10:46:23.5900820Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221201104555.xml 2022-12-01T10:46:23.5901758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5902301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5903065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5903654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5904208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtd94x_t 2022-12-01T10:46:23.5904964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtd94x_t/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5905774Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5906115Z 2022-12-01T10:46:23.5906234Z Running tests... 2022-12-01T10:46:23.5906703Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5907298Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-12-01T10:46:23.5907759Z 2022-12-01T10:46:23.5908072Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5908432Z Ran 1 test in 0.001s 2022-12-01T10:46:23.5908609Z 2022-12-01T10:46:23.5908714Z OK (skipped=1) 2022-12-01T10:46:23.5908873Z 2022-12-01T10:46:23.5909005Z Generating XML reports... 2022-12-01T10:46:23.5909739Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221201104557.xml 2022-12-01T10:46:23.5910522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5911015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5911702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5912236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5912817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16_1q9wq 2022-12-01T10:46:23.5913452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16_1q9wq/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5914218Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5914537Z 2022-12-01T10:46:23.5914633Z Running tests... 2022-12-01T10:46:23.5915159Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5915797Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-12-01T10:46:23.5916179Z 2022-12-01T10:46:23.5916533Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5916931Z Ran 1 test in 0.001s 2022-12-01T10:46:23.5917121Z 2022-12-01T10:46:23.5917255Z OK (skipped=1) 2022-12-01T10:46:23.5917460Z 2022-12-01T10:46:23.5917600Z Generating XML reports... 2022-12-01T10:46:23.5918521Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221201104558.xml 2022-12-01T10:46:23.5919509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5920090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5920828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5921385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5921965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf157mwnt 2022-12-01T10:46:23.5922644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf157mwnt/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5923425Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5923747Z 2022-12-01T10:46:23.5923866Z Running tests... 2022-12-01T10:46:23.5924407Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5925046Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-12-01T10:46:23.5925420Z 2022-12-01T10:46:23.5925751Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5926354Z Ran 1 test in 0.001s 2022-12-01T10:46:23.5926549Z 2022-12-01T10:46:23.5926669Z OK (skipped=1) 2022-12-01T10:46:23.5926857Z 2022-12-01T10:46:23.5926994Z Generating XML reports... 2022-12-01T10:46:23.5927814Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221201104559.xml 2022-12-01T10:46:23.5928784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5929469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5930238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5930857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5931461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s5ml_85 2022-12-01T10:46:23.5932207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s5ml_85/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5933027Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5933379Z 2022-12-01T10:46:23.5933510Z Running tests... 2022-12-01T10:46:23.5934055Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5934777Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13516 2022-12-01T10:46:23.5935485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13517 2022-12-01T10:46:23.5936303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5936861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5937601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5938229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5938865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp767j49od 2022-12-01T10:46:23.5939569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp767j49od/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5940461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5941051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5941834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5942440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5943037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9iumfgb 2022-12-01T10:46:23.5943763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9iumfgb/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5944433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:23.5945034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:23.5945525Z skip: Need at least 2 CUDA devices (1.522s) 2022-12-01T10:46:23.5945771Z 2022-12-01T10:46:23.5946137Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5946540Z Ran 1 test in 1.522s 2022-12-01T10:46:23.5946725Z 2022-12-01T10:46:23.5946842Z OK (skipped=1) 2022-12-01T10:46:23.5947024Z 2022-12-01T10:46:23.5947172Z Generating XML reports... 2022-12-01T10:46:23.5948028Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104601.xml 2022-12-01T10:46:23.5949097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5949683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5950470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5951090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5951720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21djov1g 2022-12-01T10:46:23.5952499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21djov1g/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5953336Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5953709Z 2022-12-01T10:46:23.5953824Z Running tests... 2022-12-01T10:46:23.5954368Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5955068Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13549 2022-12-01T10:46:23.5955778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13550 2022-12-01T10:46:23.5956618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5957194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5957970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5958737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5959355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1syo4mkp 2022-12-01T10:46:23.5960051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1syo4mkp/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5960962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5961518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5962263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5962875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5963445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5nn5cfj 2022-12-01T10:46:23.5964102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5nn5cfj/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5964729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:23.5965373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:23.5965883Z skip: Need at least 2 CUDA devices (1.522s) 2022-12-01T10:46:23.5966182Z 2022-12-01T10:46:23.5966564Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5966963Z Ran 1 test in 1.522s 2022-12-01T10:46:23.5967154Z 2022-12-01T10:46:23.5967268Z OK (skipped=1) 2022-12-01T10:46:23.5967450Z 2022-12-01T10:46:23.5967591Z Generating XML reports... 2022-12-01T10:46:23.5968451Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104604.xml 2022-12-01T10:46:23.5969456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5970041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5970816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5971442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5972027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rvxf5xd 2022-12-01T10:46:23.5973026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rvxf5xd/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5973893Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5974235Z 2022-12-01T10:46:23.5974356Z Running tests... 2022-12-01T10:46:23.5974856Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5975548Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13582 2022-12-01T10:46:23.5976373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13583 2022-12-01T10:46:23.5977196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5977764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5978563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5979160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5979723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpba8stg9p 2022-12-01T10:46:23.5980445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpba8stg9p/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5981350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5981935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5982783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5983390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5984004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw1s254y 2022-12-01T10:46:23.5984726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw1s254y/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5985401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:23.5986037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:23.5986553Z skip: Need at least 2 CUDA devices (1.522s) 2022-12-01T10:46:23.5986789Z 2022-12-01T10:46:23.5987138Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5987550Z Ran 1 test in 1.523s 2022-12-01T10:46:23.5987742Z 2022-12-01T10:46:23.5987863Z OK (skipped=1) 2022-12-01T10:46:23.5988046Z 2022-12-01T10:46:23.5988184Z Generating XML reports... 2022-12-01T10:46:23.5989056Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104607.xml 2022-12-01T10:46:23.5990095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5990667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5991415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5992044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.5992639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgdkve4a 2022-12-01T10:46:23.5993329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgdkve4a/_remote_module_non_scriptable.py 2022-12-01T10:46:23.5994121Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.5994464Z 2022-12-01T10:46:23.5994588Z Running tests... 2022-12-01T10:46:23.5995120Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.5995901Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13615 2022-12-01T10:46:23.5996621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13616 2022-12-01T10:46:23.5997447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.5998052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.5999078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.5999674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6000283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmegxke0 2022-12-01T10:46:23.6000975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmegxke0/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6001868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6002459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6003234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6003839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6004453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmhavrkw 2022-12-01T10:46:23.6005211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmhavrkw/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6005881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:23.6006548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:23.6007051Z skip: Need at least 2 CUDA devices (1.522s) 2022-12-01T10:46:23.6007305Z 2022-12-01T10:46:23.6007678Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6008094Z Ran 1 test in 1.522s 2022-12-01T10:46:23.6008297Z 2022-12-01T10:46:23.6008420Z OK (skipped=1) 2022-12-01T10:46:23.6008601Z 2022-12-01T10:46:23.6008740Z Generating XML reports... 2022-12-01T10:46:23.6009605Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104609.xml 2022-12-01T10:46:23.6010600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6011206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6012008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6012636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6013211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gw9tfvu 2022-12-01T10:46:23.6013917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gw9tfvu/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6014758Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.6015111Z 2022-12-01T10:46:23.6015225Z Running tests... 2022-12-01T10:46:23.6015776Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6016491Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13648 2022-12-01T10:46:23.6017223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13649 2022-12-01T10:46:23.6018046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6018643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6019567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6020182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6020817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplj0to911 2022-12-01T10:46:23.6021672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6022327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6023082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6023706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6024315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsz715bnh 2022-12-01T10:46:23.6025042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplj0to911/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6025755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsz715bnh/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6026457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:23.6027079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:23.6027582Z skip: Need at least 2 CUDA devices (1.421s) 2022-12-01T10:46:23.6027825Z 2022-12-01T10:46:23.6028197Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6028602Z Ran 1 test in 1.422s 2022-12-01T10:46:23.6028793Z 2022-12-01T10:46:23.6028914Z OK (skipped=1) 2022-12-01T10:46:23.6029085Z 2022-12-01T10:46:23.6029225Z Generating XML reports... 2022-12-01T10:46:23.6030113Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104612.xml 2022-12-01T10:46:23.6031099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6031654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6032395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6032977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6033570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzayotjnk 2022-12-01T10:46:23.6034299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzayotjnk/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6035116Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.6035483Z 2022-12-01T10:46:23.6035595Z Running tests... 2022-12-01T10:46:23.6036082Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6036777Z test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13681 2022-12-01T10:46:23.6037478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13682 2022-12-01T10:46:23.6038394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6038920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6039668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6040285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6040910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbw9n93pp 2022-12-01T10:46:23.6041703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbw9n93pp/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6042589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6043164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6043913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6044512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6045211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd3pjn8e 2022-12-01T10:46:23.6045886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd3pjn8e/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6046582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:23.6047223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:23.6047737Z skip: Need at least 2 CUDA devices (1.523s) 2022-12-01T10:46:23.6047968Z 2022-12-01T10:46:23.6048327Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6048722Z Ran 1 test in 1.523s 2022-12-01T10:46:23.6048909Z 2022-12-01T10:46:23.6049028Z OK (skipped=1) 2022-12-01T10:46:23.6049209Z 2022-12-01T10:46:23.6049352Z Generating XML reports... 2022-12-01T10:46:23.6050229Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104615.xml 2022-12-01T10:46:23.6051246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6051819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6052575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6053187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6053790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67bvrtw6 2022-12-01T10:46:23.6054505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67bvrtw6/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6055324Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.6055668Z 2022-12-01T10:46:23.6055794Z Running tests... 2022-12-01T10:46:23.6056334Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6057027Z test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13714 2022-12-01T10:46:23.6057714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13715 2022-12-01T10:46:23.6058543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6059137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6059875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6060494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6061109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwh6v3eq 2022-12-01T10:46:23.6061845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwh6v3eq/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6062761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6063332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6064102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6064823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6065422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwkyg152 2022-12-01T10:46:23.6066155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwkyg152/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6066798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:23.6067403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:23.6067951Z skip: Need at least 2 CUDA devices (1.522s) 2022-12-01T10:46:23.6068188Z 2022-12-01T10:46:23.6068568Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6068991Z Ran 1 test in 1.522s 2022-12-01T10:46:23.6069170Z 2022-12-01T10:46:23.6069299Z OK (skipped=1) 2022-12-01T10:46:23.6069487Z 2022-12-01T10:46:23.6069638Z Generating XML reports... 2022-12-01T10:46:23.6070535Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104618.xml 2022-12-01T10:46:23.6071522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6072094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6072871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6073480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6074094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8gjmc7w 2022-12-01T10:46:23.6074839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8gjmc7w/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6075701Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-12-01T10:46:23.6076078Z 2022-12-01T10:46:23.6076199Z Running tests... 2022-12-01T10:46:23.6076737Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6077391Z test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13747 2022-12-01T10:46:23.6078083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13748 2022-12-01T10:46:23.6078955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6079442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6080212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6080836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6081414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6z8uznq2 2022-12-01T10:46:23.6082109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6z8uznq2/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6082983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:23.6083538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:23.6084313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:23.6084935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:23.6085534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_h03f1r6 2022-12-01T10:46:23.6086281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_h03f1r6/_remote_module_non_scriptable.py 2022-12-01T10:46:23.6086932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:23.6087629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:23.6088113Z skip: Need at least 2 CUDA devices (1.423s) 2022-12-01T10:46:23.6088345Z 2022-12-01T10:46:23.6088721Z ---------------------------------------------------------------------- 2022-12-01T10:46:23.6089136Z Ran 1 test in 1.423s 2022-12-01T10:46:23.6089329Z 2022-12-01T10:46:23.6089450Z OK (skipped=1) 2022-12-01T10:46:23.6089617Z 2022-12-01T10:46:23.6089759Z Generating XML reports... 2022-12-01T10:46:23.6090634Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104621.xml 2022-12-01T10:46:23.6091274Z 2022-12-01T10:46:23.6091763Z ##[endgroup] 2022-12-01T10:46:23.6092530Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_r5vhj01r) 2022-12-01T10:46:23.6092971Z 2022-12-01T10:46:23.6093351Z Running distributed/algorithms/test_join ... [2022-12-01 10:46:23.586882] 2022-12-01T10:46:23.6094288Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:23.587222] 2022-12-01T10:46:35.9287072Z 2022-12-01T10:46:35.9288085Z Expand the folded group to see the log file of distributed/algorithms/test_join 2022-12-01T10:46:35.9288858Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_gz_42f1j) 2022-12-01T10:46:35.9289505Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join 2022-12-01T10:46:35.9289721Z 2022-12-01T10:46:35.9289796Z Running tests... 2022-12-01T10:46:35.9290245Z ---------------------------------------------------------------------- 2022-12-01T10:46:35.9293630Z test_join_kwargs (__main__.TestJoin) 2022-12-01T10:46:35.9294433Z Tests passing keyword arguments to the context manager. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13780 2022-12-01T10:46:35.9295398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13781 2022-12-01T10:46:35.9296302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9296871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9297701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9298414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9298927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9299708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9300317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9301122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9301755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9302323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9302986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9311503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9312403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9313179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9313632Z ok (1.443s) 2022-12-01T10:46:35.9313953Z test_multiple_joinable_disable (__main__.TestJoin) 2022-12-01T10:46:35.9314791Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13808 2022-12-01T10:46:35.9315413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13809 2022-12-01T10:46:35.9316179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9316716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9317558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9318168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9321958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9322801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9323352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9324126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9324699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9325208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9325810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9326511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9327274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9328097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9328542Z ok (1.211s) 2022-12-01T10:46:35.9328865Z test_multiple_joinables (__main__.TestJoin) 2022-12-01T10:46:35.9329612Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13836 2022-12-01T10:46:35.9330249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13837 2022-12-01T10:46:35.9331001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9331533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9332283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9332905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9333453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9334139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9334515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9334952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9335301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9335620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9335985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9336353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9359487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9360576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9361126Z ok (1.211s) 2022-12-01T10:46:35.9361510Z test_multiple_joinables_throw (__main__.TestJoin) 2022-12-01T10:46:35.9362123Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13864 2022-12-01T10:46:35.9362531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13865 2022-12-01T10:46:35.9363431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9363954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9364476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9364833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9365169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9365634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9365953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9366450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9366805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9367134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9367487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9367860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9368364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9368871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9369165Z ok (1.210s) 2022-12-01T10:46:35.9369364Z test_single_joinable (__main__.TestJoin) 2022-12-01T10:46:35.9369825Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13892 2022-12-01T10:46:35.9370208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13893 2022-12-01T10:46:35.9370665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9370998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9371414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9371762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9372086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9372548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9372861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9373290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9373637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9373960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9374308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9374735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9375236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9375745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9376038Z ok (1.210s) 2022-12-01T10:46:35.9376246Z test_single_joinable_disable (__main__.TestJoin) 2022-12-01T10:46:35.9376637Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13920 2022-12-01T10:46:35.9377008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13921 2022-12-01T10:46:35.9377459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9377789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9378210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9378557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9378905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9379367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9379700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9380116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9380461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9380783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9381135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9381502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9381997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9382517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9382792Z ok (1.210s) 2022-12-01T10:46:35.9383004Z test_single_joinable_main_hooks (__main__.TestJoin) 2022-12-01T10:46:35.9383355Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13948 2022-12-01T10:46:35.9383721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13949 2022-12-01T10:46:35.9384174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9384504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9384930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9385264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9385586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9386047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9386375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9386785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9387131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9387484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9387834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9388202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9388693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9389244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9389521Z ok (1.210s) 2022-12-01T10:46:35.9389732Z test_single_joinable_post_hooks (__main__.TestJoin) 2022-12-01T10:46:35.9390184Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13976 2022-12-01T10:46:35.9390558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13977 2022-12-01T10:46:35.9391016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9391347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9391774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9392108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9392433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9392890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9393216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9393627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9393973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9394296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9394645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9395013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9395513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9396033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9396307Z ok (1.211s) 2022-12-01T10:46:35.9399624Z test_single_joinable_throw (__main__.TestJoin) 2022-12-01T10:46:35.9400304Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14004 2022-12-01T10:46:35.9400873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14005 2022-12-01T10:46:35.9401645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9402247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9402676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9403034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9403362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:35.9403812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:35.9404147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:35.9404766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:35.9405121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:35.9405434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:35.9405803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:35.9406242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:35.9406786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9407311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:35.9407604Z ok (1.210s) 2022-12-01T10:46:35.9407710Z 2022-12-01T10:46:35.9407914Z ---------------------------------------------------------------------- 2022-12-01T10:46:35.9408145Z Ran 9 tests in 11.127s 2022-12-01T10:46:35.9408260Z 2022-12-01T10:46:35.9408323Z OK 2022-12-01T10:46:35.9408415Z 2022-12-01T10:46:35.9408502Z Generating XML reports... 2022-12-01T10:46:35.9408922Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221201104624.xml 2022-12-01T10:46:35.9409167Z 2022-12-01T10:46:35.9409589Z ##[endgroup] 2022-12-01T10:46:35.9410049Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_gz_42f1j) 2022-12-01T10:46:35.9410315Z 2022-12-01T10:46:35.9410525Z Running distributed/test_c10d_object_collectives ... [2022-12-01 10:46:35.928934] 2022-12-01T10:46:35.9411064Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:35.929310] 2022-12-01T10:46:42.2135964Z 2022-12-01T10:46:42.2136574Z Expand the folded group to see the log file of distributed/test_c10d_object_collectives 2022-12-01T10:46:42.2137695Z ##[group]PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_h6kenaff) 2022-12-01T10:46:42.2138862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives 2022-12-01T10:46:42.2139250Z 2022-12-01T10:46:42.2139395Z Running tests... 2022-12-01T10:46:42.2139948Z ---------------------------------------------------------------------- 2022-12-01T10:46:42.2140667Z test_all_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14043 2022-12-01T10:46:42.2141341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14044 2022-12-01T10:46:42.2141851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:42.2142192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:42.2142624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:42.2142992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:42.2143462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:42.2143987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:42.2144391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:42.2145201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:42.2145559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:42.2146074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:42.2146579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:42.2147078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:42.2147669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:42.2148347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:42.2148625Z ok (1.439s) 2022-12-01T10:46:42.2148957Z test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14071 2022-12-01T10:46:42.2149360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14072 2022-12-01T10:46:42.2149808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:42.2150142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:42.2150573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:42.2150919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:42.2151229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:42.2151694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:42.2152025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:42.2152443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:42.2152792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:42.2153119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:42.2153484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:42.2153839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:42.2154336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:42.2154855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:42.2155142Z ok (1.210s) 2022-12-01T10:46:42.2155447Z test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14099 2022-12-01T10:46:42.2155840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14100 2022-12-01T10:46:42.2156297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:42.2156614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:42.2157041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:42.2157386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:42.2157710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:42.2158160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:42.2158692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:42.2159125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:42.2159527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:42.2159858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:42.2160223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:42.2160596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:42.2161082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:42.2161648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:42.2161939Z ok (1.210s) 2022-12-01T10:46:42.2162264Z test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14127 2022-12-01T10:46:42.2162644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14128 2022-12-01T10:46:42.2163104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:42.2163438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:42.2163851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:42.2164201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:42.2164527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-12-01T10:46:42.2164988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests 2022-12-01T10:46:42.2165301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-12-01T10:46:42.2165728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests 2022-12-01T10:46:42.2166159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-12-01T10:46:42.2166485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-12-01T10:46:42.2166838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-12-01T10:46:42.2167209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-12-01T10:46:42.2167712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:42.2168220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-12-01T10:46:42.2168509Z ok (1.210s) 2022-12-01T10:46:42.2168618Z 2022-12-01T10:46:42.2168820Z ---------------------------------------------------------------------- 2022-12-01T10:46:42.2169064Z Ran 4 tests in 5.070s 2022-12-01T10:46:42.2169165Z 2022-12-01T10:46:42.2169230Z OK 2022-12-01T10:46:42.2169323Z 2022-12-01T10:46:42.2169412Z Generating XML reports... 2022-12-01T10:46:42.2169888Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221201104636.xml 2022-12-01T10:46:42.2170161Z 2022-12-01T10:46:42.2170438Z ##[endgroup] 2022-12-01T10:46:42.2170915Z FINISHED PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_h6kenaff) 2022-12-01T10:46:42.2171192Z 2022-12-01T10:46:42.2171418Z Running distributed/elastic/timer/local_timer_test ... [2022-12-01 10:46:42.214258] 2022-12-01T10:46:42.2171962Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:42.214592] 2022-12-01T10:46:46.9018431Z 2022-12-01T10:46:46.9019609Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_test 2022-12-01T10:46:46.9020436Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_criazdo3) 2022-12-01T10:46:46.9021096Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test 2022-12-01T10:46:46.9021324Z 2022-12-01T10:46:46.9021399Z Running tests... 2022-12-01T10:46:46.9021698Z ---------------------------------------------------------------------- 2022-12-01T10:46:46.9022074Z test_acquire_release (__main__.LocalTimerServerTest) 2022-12-01T10:46:46.9022963Z tests that: ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/87154 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.228s) 2022-12-01T10:46:46.9023677Z test_expired_timers (__main__.LocalTimerServerTest) 2022-12-01T10:46:46.9023968Z tests that a single expired timer on a process should terminate ... ok (0.003s) 2022-12-01T10:46:46.9024314Z test_valid_timers (__main__.LocalTimerServerTest) 2022-12-01T10:46:46.9024627Z tests that valid timers are processed correctly and the process is left alone ... ok (0.002s) 2022-12-01T10:46:46.9024939Z test_watchdog_call_count (__main__.LocalTimerServerTest) 2022-12-01T10:46:46.9025291Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.103s) 2022-12-01T10:46:46.9025589Z test_watchdog_empty_queue (__main__.LocalTimerServerTest) 2022-12-01T10:46:46.9025870Z checks that the watchdog can run on an empty queue ... ok (0.011s) 2022-12-01T10:46:46.9026141Z test_client_interaction (__main__.LocalTimerTest) ... ok (0.006s) 2022-12-01T10:46:46.9026432Z test_exception_propagation (__main__.LocalTimerTest) ... ok (0.012s) 2022-12-01T10:46:46.9026763Z test_get_timer_recursive (__main__.LocalTimerTest) 2022-12-01T10:46:46.9027062Z If a function acquires a countdown timer with default scope, ... ok (1.361s) 2022-12-01T10:46:46.9027423Z test_happy_path (__main__.LocalTimerTest) ... ok (0.103s) 2022-12-01T10:46:46.9027686Z test_no_client (__main__.LocalTimerTest) ... ok (0.011s) 2022-12-01T10:46:46.9027937Z test_timer (__main__.LocalTimerTest) ... ok (0.126s) 2022-12-01T10:46:46.9028256Z test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.023s) 2022-12-01T10:46:46.9028579Z test_get_less_than_size (__main__.MultiprocessingRequestQueueTest) 2022-12-01T10:46:46.9028848Z Tests slow producer. ... ok (0.506s) 2022-12-01T10:46:46.9029095Z test_get_size (__main__.MultiprocessingRequestQueueTest) 2022-12-01T10:46:46.9029392Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.909s) 2022-12-01T10:46:46.9029563Z 2022-12-01T10:46:46.9029766Z ---------------------------------------------------------------------- 2022-12-01T10:46:46.9030011Z Ran 14 tests in 3.407s 2022-12-01T10:46:46.9030112Z 2022-12-01T10:46:46.9030186Z OK (skipped=1) 2022-12-01T10:46:46.9030294Z 2022-12-01T10:46:46.9030379Z Generating XML reports... 2022-12-01T10:46:46.9030868Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20221201104643.xml 2022-12-01T10:46:46.9031477Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20221201104643.xml 2022-12-01T10:46:46.9032132Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20221201104643.xml 2022-12-01T10:46:46.9032452Z 2022-12-01T10:46:46.9032712Z ##[endgroup] 2022-12-01T10:46:46.9033207Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_criazdo3) 2022-12-01T10:46:46.9033479Z 2022-12-01T10:46:46.9033691Z Running distributed/elastic/utils/util_test ... [2022-12-01 10:46:46.902202] 2022-12-01T10:46:46.9034292Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:46.902615] 2022-12-01T10:46:48.3609483Z 2022-12-01T10:46:48.3610092Z Expand the folded group to see the log file of distributed/elastic/utils/util_test 2022-12-01T10:46:48.3611050Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_ehe6w0v7) 2022-12-01T10:46:48.3611860Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test 2022-12-01T10:46:48.3612076Z 2022-12-01T10:46:48.3612137Z Running tests... 2022-12-01T10:46:48.3612443Z ---------------------------------------------------------------------- 2022-12-01T10:46:48.3612733Z test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (0.225s) 2022-12-01T10:46:48.3612988Z test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s) 2022-12-01T10:46:48.3613307Z test_synchronize (__main__.StoreUtilTest) ... ok (0.002s) 2022-12-01T10:46:48.3613593Z test_get_logger (__main__.UtilTest) ... ok (0.064s) 2022-12-01T10:46:48.3613847Z test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s) 2022-12-01T10:46:48.3614100Z test_get_logger_different (__main__.UtilTest) ... ok (0.001s) 2022-12-01T10:46:48.3614421Z test_get_logger_none (__main__.UtilTest) ... ok (0.001s) 2022-12-01T10:46:48.3614568Z 2022-12-01T10:46:48.3614769Z ---------------------------------------------------------------------- 2022-12-01T10:46:48.3615032Z Ran 7 tests in 0.295s 2022-12-01T10:46:48.3615161Z 2022-12-01T10:46:48.3615221Z OK 2022-12-01T10:46:48.3615311Z 2022-12-01T10:46:48.3615394Z Generating XML reports... 2022-12-01T10:46:48.3615906Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221201104647.xml 2022-12-01T10:46:48.3616522Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221201104647.xml 2022-12-01T10:46:48.3616820Z 2022-12-01T10:46:48.3617052Z ##[endgroup] 2022-12-01T10:46:48.3617516Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_ehe6w0v7) 2022-12-01T10:46:48.3617771Z 2022-12-01T10:46:48.3617985Z Running distributed/elastic/metrics/api_test ... [2022-12-01 10:46:48.361253] 2022-12-01T10:46:48.3618514Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:48.361584] 2022-12-01T10:46:49.7245344Z 2022-12-01T10:46:49.7245972Z Expand the folded group to see the log file of distributed/elastic/metrics/api_test 2022-12-01T10:46:49.7246887Z ##[group]PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_qa7tb6f3) 2022-12-01T10:46:49.7247563Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test 2022-12-01T10:46:49.7247773Z 2022-12-01T10:46:49.7247847Z Running tests... 2022-12-01T10:46:49.7248154Z ---------------------------------------------------------------------- 2022-12-01T10:46:49.7248447Z test_get_metric_name (__main__.MetricsApiTest) ... ok (0.225s) 2022-12-01T10:46:49.7248791Z test_inheritance (__main__.MetricsApiTest) ... ok (0.001s) 2022-12-01T10:46:49.7249069Z test_profile (__main__.MetricsApiTest) ... ok (0.001s) 2022-12-01T10:46:49.7249214Z 2022-12-01T10:46:49.7249414Z ---------------------------------------------------------------------- 2022-12-01T10:46:49.7249688Z Ran 3 tests in 0.228s 2022-12-01T10:46:49.7249789Z 2022-12-01T10:46:49.7249849Z OK 2022-12-01T10:46:49.7249941Z 2022-12-01T10:46:49.7250024Z Generating XML reports... 2022-12-01T10:46:49.7250489Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221201104649.xml 2022-12-01T10:46:49.7250928Z 2022-12-01T10:46:49.7251151Z ##[endgroup] 2022-12-01T10:46:49.7251664Z FINISHED PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_qa7tb6f3) 2022-12-01T10:46:49.7251984Z 2022-12-01T10:46:49.7252204Z Running distributed/test_launcher ... [2022-12-01 10:46:49.724780] 2022-12-01T10:46:49.7252901Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:49.725102] 2022-12-01T10:46:51.3335424Z 2022-12-01T10:46:51.3335913Z Expand the folded group to see the log file of distributed/test_launcher 2022-12-01T10:46:51.3337003Z ##[group]PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_m9b22d2i) 2022-12-01T10:46:51.3338022Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher 2022-12-01T10:46:51.3338373Z 2022-12-01T10:46:51.3338512Z Running tests... 2022-12-01T10:46:51.3338927Z ---------------------------------------------------------------------- 2022-12-01T10:46:51.3340223Z test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.223s) 2022-12-01T10:46:51.3340704Z 2022-12-01T10:46:51.3340901Z ---------------------------------------------------------------------- 2022-12-01T10:46:51.3341174Z Ran 1 test in 0.224s 2022-12-01T10:46:51.3341272Z 2022-12-01T10:46:51.3341346Z OK (skipped=1) 2022-12-01T10:46:51.3341461Z 2022-12-01T10:46:51.3341547Z Generating XML reports... 2022-12-01T10:46:51.3341999Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20221201104650.xml 2022-12-01T10:46:51.3342263Z 2022-12-01T10:46:51.3342476Z ##[endgroup] 2022-12-01T10:46:51.3342896Z FINISHED PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_m9b22d2i) 2022-12-01T10:46:51.3343132Z 2022-12-01T10:46:51.3343389Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-12-01 10:46:51.333907] 2022-12-01T10:46:51.3344959Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:51.334299] 2022-12-01T10:46:52.4347570Z 2022-12-01T10:46:52.4348173Z Expand the folded group to see the log file of distributed/_shard/sharding_plan/test_sharding_plan 2022-12-01T10:46:52.4349465Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_kw49ui0x) 2022-12-01T10:46:52.4350441Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan 2022-12-01T10:46:52.4350668Z 2022-12-01T10:46:52.4350729Z Running tests... 2022-12-01T10:46:52.4351042Z ---------------------------------------------------------------------- 2022-12-01T10:46:52.4351405Z test_custom_sharding_planner (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:52.4351891Z test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:52.4352294Z test_shard_module_sub_process_group (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:52.4352684Z test_sharding_plan_errors (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:52.4353247Z test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-12-01T10:46:52.4353465Z 2022-12-01T10:46:52.4353672Z ---------------------------------------------------------------------- 2022-12-01T10:46:52.4353903Z Ran 5 tests in 0.008s 2022-12-01T10:46:52.4354019Z 2022-12-01T10:46:52.4354092Z OK (skipped=5) 2022-12-01T10:46:52.4354200Z 2022-12-01T10:46:52.4354299Z Generating XML reports... 2022-12-01T10:46:52.4354925Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221201104652.xml 2022-12-01T10:46:52.4355278Z 2022-12-01T10:46:52.4355514Z ##[endgroup] 2022-12-01T10:46:52.4356029Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_kw49ui0x) 2022-12-01T10:46:52.4356322Z 2022-12-01T10:46:52.4356560Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-12-01 10:46:52.435055] 2022-12-01T10:46:52.4357116Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:52.435371] 2022-12-01T10:46:53.5631540Z 2022-12-01T10:46:53.5632123Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-12-01T10:46:53.5633509Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_0cc8crgn) 2022-12-01T10:46:53.5634238Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops 2022-12-01T10:46:53.5634462Z 2022-12-01T10:46:53.5634539Z Running tests... 2022-12-01T10:46:53.5634849Z ---------------------------------------------------------------------- 2022-12-01T10:46:53.5635174Z test_clone (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:53.5635602Z test_deep_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:53.5635952Z test_detach (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:53.5636307Z test_inplace_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:53.5636658Z test_set_requires_grad (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:53.5636860Z 2022-12-01T10:46:53.5637058Z ---------------------------------------------------------------------- 2022-12-01T10:46:53.5637297Z Ran 5 tests in 0.003s 2022-12-01T10:46:53.5637411Z 2022-12-01T10:46:53.5637470Z OK (skipped=5) 2022-12-01T10:46:53.5637578Z 2022-12-01T10:46:53.5637663Z Generating XML reports... 2022-12-01T10:46:53.5638143Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221201104653.xml 2022-12-01T10:46:53.5638600Z 2022-12-01T10:46:53.5638904Z ##[endgroup] 2022-12-01T10:46:53.5639579Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_0cc8crgn) 2022-12-01T10:46:53.5639879Z 2022-12-01T10:46:53.5640097Z Running distributed/_shard/test_partial_tensor ... [2022-12-01 10:46:53.563441] 2022-12-01T10:46:53.5640638Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:53.563797] 2022-12-01T10:46:54.6928459Z 2022-12-01T10:46:54.6929005Z Expand the folded group to see the log file of distributed/_shard/test_partial_tensor 2022-12-01T10:46:54.6930162Z ##[group]PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_ww7hv38b) 2022-12-01T10:46:54.6931501Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor 2022-12-01T10:46:54.6931746Z 2022-12-01T10:46:54.6931829Z Running tests... 2022-12-01T10:46:54.6932233Z ---------------------------------------------------------------------- 2022-12-01T10:46:54.6932630Z test_cat (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:54.6933107Z test_cat_errors (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:54.6933579Z test_transpose (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:46:54.6933970Z test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:54.6934399Z test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:54.6934631Z 2022-12-01T10:46:54.6934836Z ---------------------------------------------------------------------- 2022-12-01T10:46:54.6935087Z Ran 5 tests in 0.004s 2022-12-01T10:46:54.6935246Z 2022-12-01T10:46:54.6935358Z OK (skipped=5) 2022-12-01T10:46:54.6935475Z 2022-12-01T10:46:54.6935569Z Generating XML reports... 2022-12-01T10:46:54.6936147Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20221201104654.xml 2022-12-01T10:46:54.6936769Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20221201104654.xml 2022-12-01T10:46:54.6937034Z 2022-12-01T10:46:54.6937278Z ##[endgroup] 2022-12-01T10:46:54.6937745Z FINISHED PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_ww7hv38b) 2022-12-01T10:46:54.6938009Z 2022-12-01T10:46:54.6938243Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-12-01 10:46:54.693153] 2022-12-01T10:46:54.6938777Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:54.693474] 2022-12-01T10:46:55.8004442Z 2022-12-01T10:46:55.8005034Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_init 2022-12-01T10:46:55.8006270Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_37p0v25o) 2022-12-01T10:46:55.8006954Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init 2022-12-01T10:46:55.8007172Z 2022-12-01T10:46:55.8007233Z Running tests... 2022-12-01T10:46:55.8007544Z ---------------------------------------------------------------------- 2022-12-01T10:46:55.8007876Z test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit) 2022-12-01T10:46:55.8008316Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:55.8008743Z test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit) 2022-12-01T10:46:55.8009199Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:55.8009598Z test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit) 2022-12-01T10:46:55.8009939Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:55.8010199Z 2022-12-01T10:46:55.8010406Z ---------------------------------------------------------------------- 2022-12-01T10:46:55.8010651Z Ran 3 tests in 0.003s 2022-12-01T10:46:55.8010762Z 2022-12-01T10:46:55.8010839Z OK (skipped=3) 2022-12-01T10:46:55.8010990Z 2022-12-01T10:46:55.8011238Z Generating XML reports... 2022-12-01T10:46:55.8011963Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221201104655.xml 2022-12-01T10:46:55.8012252Z 2022-12-01T10:46:55.8012502Z ##[endgroup] 2022-12-01T10:46:55.8013003Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_37p0v25o) 2022-12-01T10:46:55.8013354Z 2022-12-01T10:46:55.8013584Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-12-01 10:46:55.800746] 2022-12-01T10:46:55.8014145Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:55.801069] 2022-12-01T10:46:56.9045108Z 2022-12-01T10:46:56.9045653Z Expand the folded group to see the log file of distributed/_shard/sharded_optim/test_sharded_optim 2022-12-01T10:46:56.9047646Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_drm5rmeq) 2022-12-01T10:46:56.9048370Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim 2022-12-01T10:46:56.9048592Z 2022-12-01T10:46:56.9048692Z Running tests... 2022-12-01T10:46:56.9049031Z ---------------------------------------------------------------------- 2022-12-01T10:46:56.9049417Z test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:56.9049890Z test_sharded_optim (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:56.9050095Z 2022-12-01T10:46:56.9050296Z ---------------------------------------------------------------------- 2022-12-01T10:46:56.9050534Z Ran 2 tests in 0.003s 2022-12-01T10:46:56.9050650Z 2022-12-01T10:46:56.9050714Z OK (skipped=2) 2022-12-01T10:46:56.9050820Z 2022-12-01T10:46:56.9050904Z Generating XML reports... 2022-12-01T10:46:56.9051406Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20221201104656.xml 2022-12-01T10:46:56.9051688Z 2022-12-01T10:46:56.9051944Z ##[endgroup] 2022-12-01T10:46:56.9052456Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_drm5rmeq) 2022-12-01T10:46:56.9052796Z 2022-12-01T10:46:56.9053030Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-12-01 10:46:56.904780] 2022-12-01T10:46:56.9053604Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:56.905104] 2022-12-01T10:46:58.0099206Z 2022-12-01T10:46:58.0099847Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding 2022-12-01T10:46:58.0101246Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_2cvqnxmm) 2022-12-01T10:46:58.0101957Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding 2022-12-01T10:46:58.0102230Z 2022-12-01T10:46:58.0102307Z Running tests... 2022-12-01T10:46:58.0102638Z ---------------------------------------------------------------------- 2022-12-01T10:46:58.0103156Z test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:58.0103571Z test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:58.0103789Z 2022-12-01T10:46:58.0104175Z ---------------------------------------------------------------------- 2022-12-01T10:46:58.0104420Z Ran 2 tests in 0.002s 2022-12-01T10:46:58.0104522Z 2022-12-01T10:46:58.0104594Z OK (skipped=2) 2022-12-01T10:46:58.0104700Z 2022-12-01T10:46:58.0104785Z Generating XML reports... 2022-12-01T10:46:58.0105285Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221201104657.xml 2022-12-01T10:46:58.0105637Z 2022-12-01T10:46:58.0105872Z ##[endgroup] 2022-12-01T10:46:58.0106435Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_2cvqnxmm) 2022-12-01T10:46:58.0106778Z 2022-12-01T10:46:58.0107034Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-12-01 10:46:58.010178] 2022-12-01T10:46:58.0107626Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:58.010535] 2022-12-01T10:46:59.1101295Z 2022-12-01T10:46:59.1101831Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-12-01T10:46:59.1103107Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_z32u7oiq) 2022-12-01T10:46:59.1103886Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard 2022-12-01T10:46:59.1104114Z 2022-12-01T10:46:59.1104188Z Running tests... 2022-12-01T10:46:59.1104494Z ---------------------------------------------------------------------- 2022-12-01T10:46:59.1104834Z test_sharded_tensor_reshard (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:59.1105232Z test_sharded_tensor_reshard_errors (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-12-01T10:46:59.1105461Z 2022-12-01T10:46:59.1105700Z ---------------------------------------------------------------------- 2022-12-01T10:46:59.1105950Z Ran 2 tests in 0.002s 2022-12-01T10:46:59.1106097Z 2022-12-01T10:46:59.1106171Z OK (skipped=2) 2022-12-01T10:46:59.1106277Z 2022-12-01T10:46:59.1106362Z Generating XML reports... 2022-12-01T10:46:59.1106891Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221201104658.xml 2022-12-01T10:46:59.1107160Z 2022-12-01T10:46:59.1107367Z ##[endgroup] 2022-12-01T10:46:59.1107913Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_z32u7oiq) 2022-12-01T10:46:59.1108226Z 2022-12-01T10:46:59.1108573Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-12-01 10:46:59.110431] 2022-12-01T10:46:59.1109405Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:59.110772] 2022-12-01T10:47:00.2132907Z 2022-12-01T10:47:00.2133574Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding_bag 2022-12-01T10:47:00.2134842Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_bfop1u06) 2022-12-01T10:47:00.2135704Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag 2022-12-01T10:47:00.2135933Z 2022-12-01T10:47:00.2136006Z Running tests... 2022-12-01T10:47:00.2136357Z ---------------------------------------------------------------------- 2022-12-01T10:47:00.2136959Z test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:47:00.2137476Z test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:47:00.2137809Z 2022-12-01T10:47:00.2138002Z ---------------------------------------------------------------------- 2022-12-01T10:47:00.2138309Z Ran 2 tests in 0.001s 2022-12-01T10:47:00.2138426Z 2022-12-01T10:47:00.2138499Z OK (skipped=2) 2022-12-01T10:47:00.2138607Z 2022-12-01T10:47:00.2138693Z Generating XML reports... 2022-12-01T10:47:00.2139197Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20221201104700.xml 2022-12-01T10:47:00.2139493Z 2022-12-01T10:47:00.2139736Z ##[endgroup] 2022-12-01T10:47:00.2140423Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_bfop1u06) 2022-12-01T10:47:00.2140778Z 2022-12-01T10:47:00.2141017Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2022-12-01 10:47:00.213635] 2022-12-01T10:47:00.2141572Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:47:00.213949] 2022-12-01T10:47:01.3293341Z 2022-12-01T10:47:01.3294048Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_softmax 2022-12-01T10:47:01.3295141Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_l8g2p1yt) 2022-12-01T10:47:01.3295650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdyp2z_e 2022-12-01T10:47:01.3296075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdyp2z_e/_remote_module_non_scriptable.py 2022-12-01T10:47:01.3296678Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax 2022-12-01T10:47:01.3296897Z 2022-12-01T10:47:01.3296957Z Running tests... 2022-12-01T10:47:01.3297322Z ---------------------------------------------------------------------- 2022-12-01T10:47:01.3297681Z test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:47:01.3298216Z test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-12-01T10:47:01.3298433Z 2022-12-01T10:47:01.3298621Z ---------------------------------------------------------------------- 2022-12-01T10:47:01.3298861Z Ran 2 tests in 0.001s 2022-12-01T10:47:01.3298974Z 2022-12-01T10:47:01.3299048Z OK (skipped=2) 2022-12-01T10:47:01.3299155Z 2022-12-01T10:47:01.3299233Z Generating XML reports... 2022-12-01T10:47:01.3299721Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20221201104701.xml 2022-12-01T10:47:01.3299993Z 2022-12-01T10:47:01.3300230Z ##[endgroup] 2022-12-01T10:47:01.3300798Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_l8g2p1yt) 2022-12-01T10:47:01.3301080Z 2022-12-01T10:47:01.3301409Z Running distributed/_shard/test_replicated_tensor ... [2022-12-01 10:47:01.329670] 2022-12-01T10:47:01.3302709Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:47:01.330031] 2022-12-01T10:47:02.4389557Z 2022-12-01T10:47:02.4390043Z Expand the folded group to see the log file of distributed/_shard/test_replicated_tensor 2022-12-01T10:47:02.4391077Z ##[group]PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_42zmrhcd) 2022-12-01T10:47:02.4403537Z 2022-12-01T10:47:02.4404033Z ##[endgroup] 2022-12-01T10:47:02.4404651Z FINISHED PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_42zmrhcd) 2022-12-01T10:47:02.4405067Z 2022-12-01T10:47:02.4405277Z Running distributed/elastic/events/lib_test ... [2022-12-01 10:47:02.439187] 2022-12-01T10:47:02.4405720Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-12-01 10:47:02.439506] 2022-12-01T10:47:04.1833574Z 2022-12-01T10:47:04.1834214Z Expand the folded group to see the log file of distributed/elastic/events/lib_test 2022-12-01T10:47:04.1835095Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_6jm8x151) 2022-12-01T10:47:04.1835485Z ============================= test session starts ============================== 2022-12-01T10:47:04.1835987Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:04.1836232Z cachedir: .pytest_cache 2022-12-01T10:47:04.1836667Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:04.1837050Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:04.1837500Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:04.1837788Z collecting ... collected 8 items 2022-12-01T10:47:04.1838949Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2022-12-01T10:47:04.1839781Z 2022-12-01T10:47:04.1839949Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%] 2022-12-01T10:47:04.1840294Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%] 2022-12-01T10:47:04.1840706Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%] 2022-12-01T10:47:04.1841078Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%] 2022-12-01T10:47:04.1841531Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%] 2022-12-01T10:47:04.1841931Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%] 2022-12-01T10:47:04.1842296Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%] 2022-12-01T10:47:04.1842649Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%] 2022-12-01T10:47:04.1842841Z 2022-12-01T10:47:04.1842950Z ============================== 8 passed in 0.90s =============================== 2022-12-01T10:47:04.1843088Z 2022-12-01T10:47:04.1843313Z ##[endgroup] 2022-12-01T10:47:04.1843783Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_6jm8x151) 2022-12-01T10:47:04.1844057Z 2022-12-01T10:47:04.1844450Z Running distributed/fsdp/test_shard_utils ... [2022-12-01 10:47:04.183609] 2022-12-01T10:47:04.1844979Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:47:04.183969] 2022-12-01T10:47:05.2894236Z 2022-12-01T10:47:05.2894896Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils 2022-12-01T10:47:05.2895912Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_7gmb7ljq) 2022-12-01T10:47:05.2896185Z 2022-12-01T10:47:05.2896400Z ##[endgroup] 2022-12-01T10:47:05.2896957Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_7gmb7ljq) 2022-12-01T10:47:05.2897217Z 2022-12-01T10:47:05.2898809Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-12-01 10:47:05.289670] 2022-12-01T10:47:05.2902504Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-12-01 10:47:05.289998] 2022-12-01T10:47:06.6521997Z 2022-12-01T10:47:06.6522434Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe 2022-12-01T10:47:06.6523469Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_jxswsnjf) 2022-12-01T10:47:06.6523910Z ============================= test session starts ============================== 2022-12-01T10:47:06.6524422Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:06.6524684Z cachedir: .pytest_cache 2022-12-01T10:47:06.6525153Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:06.6525535Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:06.6525825Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:06.6526442Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:06.6526755Z collecting ... collected 13 items 2022-12-01T10:47:06.6528658Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip 2022-12-01T10:47:06.6529852Z 2022-12-01T10:47:06.6530085Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] SKIPPED [ 7%] 2022-12-01T10:47:06.6530536Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] SKIPPED [ 15%] 2022-12-01T10:47:06.6530930Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] SKIPPED [ 23%] 2022-12-01T10:47:06.6531370Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%] 2022-12-01T10:47:06.6531788Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] SKIPPED [ 38%] 2022-12-01T10:47:06.6532208Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] SKIPPED [ 46%] 2022-12-01T10:47:06.6532828Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] SKIPPED [ 53%] 2022-12-01T10:47:06.6533236Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%] 2022-12-01T10:47:06.6533693Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] SKIPPED [ 69%] 2022-12-01T10:47:06.6534153Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] SKIPPED [ 76%] 2022-12-01T10:47:06.6534551Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] SKIPPED [ 84%] 2022-12-01T10:47:06.6535077Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%] 2022-12-01T10:47:06.6535407Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED [100%] 2022-12-01T10:47:06.6535616Z 2022-12-01T10:47:06.6535750Z ======================== 1 passed, 12 skipped in 0.09s ========================= 2022-12-01T10:47:06.6535880Z 2022-12-01T10:47:06.6536122Z ##[endgroup] 2022-12-01T10:47:06.6536676Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_jxswsnjf) 2022-12-01T10:47:06.6537071Z 2022-12-01T10:47:06.6537338Z Running distributed/pipeline/sync/skip/test_leak ... [2022-12-01 10:47:06.652482] 2022-12-01T10:47:06.6537788Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-12-01 10:47:06.652820] 2022-12-01T10:47:08.1355299Z 2022-12-01T10:47:08.1355884Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_leak 2022-12-01T10:47:08.1357216Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_oaqh9985) 2022-12-01T10:47:08.1357652Z ============================= test session starts ============================== 2022-12-01T10:47:08.1358146Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:08.1358578Z cachedir: .pytest_cache 2022-12-01T10:47:08.1359004Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:08.1359332Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:08.1359619Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:08.1360039Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:08.1360459Z collecting ... collected 8 items 2022-12-01T10:47:08.1362052Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] 2022-12-01T10:47:08.1363004Z 2022-12-01T10:47:08.1363337Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%] 2022-12-01T10:47:08.1363793Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%] 2022-12-01T10:47:08.1364304Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%] 2022-12-01T10:47:08.1364761Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%] 2022-12-01T10:47:08.1365237Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%] 2022-12-01T10:47:08.1365980Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%] 2022-12-01T10:47:08.1366410Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%] 2022-12-01T10:47:08.1366751Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%] 2022-12-01T10:47:08.1366939Z 2022-12-01T10:47:08.1367071Z ============================== 8 passed in 0.20s =============================== 2022-12-01T10:47:08.1367295Z 2022-12-01T10:47:08.1367542Z ##[endgroup] 2022-12-01T10:47:08.1368079Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_oaqh9985) 2022-12-01T10:47:08.1368371Z 2022-12-01T10:47:08.1368635Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-12-01 10:47:08.135902] 2022-12-01T10:47:08.1369121Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-12-01 10:47:08.136242] 2022-12-01T10:47:09.4390045Z 2022-12-01T10:47:09.4390614Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop 2022-12-01T10:47:09.4392106Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_1dshn0mt) 2022-12-01T10:47:09.4392604Z ============================= test session starts ============================== 2022-12-01T10:47:09.4393240Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:09.4393727Z cachedir: .pytest_cache 2022-12-01T10:47:09.4394346Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:09.4394757Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:09.4395108Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:09.4395525Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:09.4395816Z collecting ... collected 7 items 2022-12-01T10:47:09.4396558Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none 2022-12-01T10:47:09.4397177Z 2022-12-01T10:47:09.4397332Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED [ 14%] 2022-12-01T10:47:09.4397652Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED [ 28%] 2022-12-01T10:47:09.4397986Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%] 2022-12-01T10:47:09.4398544Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%] 2022-12-01T10:47:09.4398889Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%] 2022-12-01T10:47:09.4399239Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%] 2022-12-01T10:47:09.4399573Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%] 2022-12-01T10:47:09.4399743Z 2022-12-01T10:47:09.4399854Z ============================== 7 passed in 0.03s =============================== 2022-12-01T10:47:09.4399994Z 2022-12-01T10:47:09.4400239Z ##[endgroup] 2022-12-01T10:47:09.4400761Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_1dshn0mt) 2022-12-01T10:47:09.4401062Z 2022-12-01T10:47:09.4401471Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-12-01 10:47:09.439322] 2022-12-01T10:47:09.4401981Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-12-01 10:47:09.439672] 2022-12-01T10:47:10.7532581Z 2022-12-01T10:47:10.7533165Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables 2022-12-01T10:47:10.7534570Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_i7xv3q76) 2022-12-01T10:47:10.7535239Z ============================= test session starts ============================== 2022-12-01T10:47:10.7535721Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:10.7535972Z cachedir: .pytest_cache 2022-12-01T10:47:10.7536410Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:10.7536734Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:10.7537074Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:10.7537521Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:10.7537809Z collecting ... collected 9 items 2022-12-01T10:47:10.7538843Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated 2022-12-01T10:47:10.7539716Z 2022-12-01T10:47:10.7539882Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%] 2022-12-01T10:47:10.7540333Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%] 2022-12-01T10:47:10.7540690Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%] 2022-12-01T10:47:10.7541024Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%] 2022-12-01T10:47:10.7541370Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%] 2022-12-01T10:47:10.7541743Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%] 2022-12-01T10:47:10.7542135Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%] 2022-12-01T10:47:10.7542491Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%] 2022-12-01T10:47:10.7542865Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%] 2022-12-01T10:47:10.7543070Z 2022-12-01T10:47:10.7543180Z ============================== 9 passed in 0.04s =============================== 2022-12-01T10:47:10.7543321Z 2022-12-01T10:47:10.7543547Z ##[endgroup] 2022-12-01T10:47:10.7544090Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_i7xv3q76) 2022-12-01T10:47:10.7544411Z 2022-12-01T10:47:10.7544829Z Running distributed/pipeline/sync/test_bugs ... [2022-12-01 10:47:10.753567] 2022-12-01T10:47:10.7545385Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-12-01 10:47:10.753906] 2022-12-01T10:47:12.2404815Z 2022-12-01T10:47:12.2405371Z Expand the folded group to see the log file of distributed/pipeline/sync/test_bugs 2022-12-01T10:47:12.2406431Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_9zzrtmhf) 2022-12-01T10:47:12.2406815Z ============================= test session starts ============================== 2022-12-01T10:47:12.2407493Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:12.2407754Z cachedir: .pytest_cache 2022-12-01T10:47:12.2408229Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:12.2408678Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:12.2408923Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:12.2409349Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:12.2409628Z collecting ... collected 4 items 2022-12-01T10:47:12.2410134Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms 2022-12-01T10:47:12.2410530Z 2022-12-01T10:47:12.2410692Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%] 2022-12-01T10:47:12.2411064Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED [ 50%] 2022-12-01T10:47:12.2411456Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait SKIPPED (2 c...) [ 75%] 2022-12-01T10:47:12.2411865Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED [100%] 2022-12-01T10:47:12.2412044Z 2022-12-01T10:47:12.2412166Z ========================= 3 passed, 1 skipped in 0.19s ========================= 2022-12-01T10:47:12.2412315Z 2022-12-01T10:47:12.2412537Z ##[endgroup] 2022-12-01T10:47:12.2413013Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_9zzrtmhf) 2022-12-01T10:47:12.2413289Z 2022-12-01T10:47:12.2413500Z Running distributed/pipeline/sync/test_copy ... [2022-12-01 10:47:12.240726] 2022-12-01T10:47:12.2413957Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-12-01 10:47:12.241058] 2022-12-01T10:47:13.5486337Z 2022-12-01T10:47:13.5486976Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy 2022-12-01T10:47:13.5487899Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_xfpj8t7o) 2022-12-01T10:47:13.5488327Z ============================= test session starts ============================== 2022-12-01T10:47:13.5488799Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:13.5489047Z cachedir: .pytest_cache 2022-12-01T10:47:13.5489480Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:13.5489809Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:13.5490074Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:13.5490581Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:13.5490917Z collecting ... collected 5 items 2022-12-01T10:47:13.5491670Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors 2022-12-01T10:47:13.5492132Z 2022-12-01T10:47:13.5492274Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED [ 20%] 2022-12-01T10:47:13.5492597Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda SKIPPED [ 40%] 2022-12-01T10:47:13.5492916Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu SKIPPED [ 60%] 2022-12-01T10:47:13.5493338Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda SKIPPED [ 80%] 2022-12-01T10:47:13.5493663Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%] 2022-12-01T10:47:13.5493894Z 2022-12-01T10:47:13.5494067Z ========================= 2 passed, 3 skipped in 0.03s ========================= 2022-12-01T10:47:13.5494213Z 2022-12-01T10:47:13.5494455Z ##[endgroup] 2022-12-01T10:47:13.5494930Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_xfpj8t7o) 2022-12-01T10:47:13.5495210Z 2022-12-01T10:47:13.5495434Z Running distributed/pipeline/sync/test_dependency ... [2022-12-01 10:47:13.548931] 2022-12-01T10:47:13.5495904Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-12-01 10:47:13.549283] 2022-12-01T10:47:14.8568185Z 2022-12-01T10:47:14.8568759Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency 2022-12-01T10:47:14.8569826Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_ll0zlidc) 2022-12-01T10:47:14.8570251Z ============================= test session starts ============================== 2022-12-01T10:47:14.8570727Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:14.8570990Z cachedir: .pytest_cache 2022-12-01T10:47:14.8571431Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:14.8571745Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:14.8572017Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:14.8572567Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:14.8572843Z collecting ... collected 6 items 2022-12-01T10:47:14.8573588Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad 2022-12-01T10:47:14.8574165Z 2022-12-01T10:47:14.8574325Z distributed/pipeline/sync/test_dependency.py::test_fork_join SKIPPED [ 16%] 2022-12-01T10:47:14.8574669Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%] 2022-12-01T10:47:14.8575016Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%] 2022-12-01T10:47:14.8575334Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED [ 66%] 2022-12-01T10:47:14.8575784Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%] 2022-12-01T10:47:14.8576156Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%] 2022-12-01T10:47:14.8576352Z 2022-12-01T10:47:14.8576463Z ========================= 5 passed, 1 skipped in 0.03s ========================= 2022-12-01T10:47:14.8576642Z 2022-12-01T10:47:14.8576866Z ##[endgroup] 2022-12-01T10:47:14.8577566Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_ll0zlidc) 2022-12-01T10:47:14.8577861Z 2022-12-01T10:47:14.8578095Z Running distributed/pipeline/sync/test_microbatch ... [2022-12-01 10:47:14.857141] 2022-12-01T10:47:14.8578552Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-12-01 10:47:14.857504] 2022-12-01T10:47:16.1735244Z 2022-12-01T10:47:16.1736050Z Expand the folded group to see the log file of distributed/pipeline/sync/test_microbatch 2022-12-01T10:47:16.1737220Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_sfv6opfa) 2022-12-01T10:47:16.1737637Z ============================= test session starts ============================== 2022-12-01T10:47:16.1738132Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:16.1738391Z cachedir: .pytest_cache 2022-12-01T10:47:16.1738930Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:16.1739274Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:16.1739533Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:16.1740002Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:16.1740350Z collecting ... collected 10 items 2022-12-01T10:47:16.1741315Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors 2022-12-01T10:47:16.1742264Z 2022-12-01T10:47:16.1742424Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED [ 10%] 2022-12-01T10:47:16.1742815Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%] 2022-12-01T10:47:16.1743151Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED [ 30%] 2022-12-01T10:47:16.1743487Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%] 2022-12-01T10:47:16.1743819Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%] 2022-12-01T10:47:16.1744151Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED [ 60%] 2022-12-01T10:47:16.1744471Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%] 2022-12-01T10:47:16.1744798Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED [ 80%] 2022-12-01T10:47:16.1745109Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%] 2022-12-01T10:47:16.1745441Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%] 2022-12-01T10:47:16.1745633Z 2022-12-01T10:47:16.1745744Z ============================== 10 passed in 0.04s ============================== 2022-12-01T10:47:16.1745883Z 2022-12-01T10:47:16.1746102Z ##[endgroup] 2022-12-01T10:47:16.1746604Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_sfv6opfa) 2022-12-01T10:47:16.1746895Z 2022-12-01T10:47:16.1747110Z Running distributed/pipeline/sync/test_pipe ... [2022-12-01 10:47:16.173816] 2022-12-01T10:47:16.1747687Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-12-01 10:47:16.174148] 2022-12-01T10:47:18.9779968Z 2022-12-01T10:47:18.9780557Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe 2022-12-01T10:47:18.9781667Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_0sa393lj) 2022-12-01T10:47:18.9782112Z ============================= test session starts ============================== 2022-12-01T10:47:18.9782777Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:18.9783030Z cachedir: .pytest_cache 2022-12-01T10:47:18.9783469Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:18.9783795Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:18.9784022Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:18.9784448Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:18.9784741Z collecting ... collected 56 items 2022-12-01T10:47:18.9789583Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper 2022-12-01T10:47:18.9794123Z 2022-12-01T10:47:18.9794294Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED [ 1%] 2022-12-01T10:47:18.9794612Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED [ 3%] 2022-12-01T10:47:18.9794978Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED [ 5%] 2022-12-01T10:47:18.9795303Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED [ 7%] 2022-12-01T10:47:18.9795661Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED [ 8%] 2022-12-01T10:47:18.9795990Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%] 2022-12-01T10:47:18.9796328Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED [ 12%] 2022-12-01T10:47:18.9796681Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED [ 14%] 2022-12-01T10:47:18.9797000Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%] 2022-12-01T10:47:18.9797402Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%] 2022-12-01T10:47:18.9797741Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED [ 19%] 2022-12-01T10:47:18.9798107Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%] 2022-12-01T10:47:18.9798629Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED [ 23%] 2022-12-01T10:47:18.9798938Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED [ 25%] 2022-12-01T10:47:18.9799318Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%] 2022-12-01T10:47:18.9799635Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED [ 28%] 2022-12-01T10:47:18.9800005Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED [ 30%] 2022-12-01T10:47:18.9800323Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%] 2022-12-01T10:47:18.9800685Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED [ 33%] 2022-12-01T10:47:18.9801004Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED [ 35%] 2022-12-01T10:47:18.9801319Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED [ 37%] 2022-12-01T10:47:18.9801686Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED [ 39%] 2022-12-01T10:47:18.9801996Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%] 2022-12-01T10:47:18.9802383Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%] 2022-12-01T10:47:18.9802725Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%] 2022-12-01T10:47:18.9803096Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%] 2022-12-01T10:47:18.9803493Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%] 2022-12-01T10:47:18.9803882Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%] 2022-12-01T10:47:18.9804220Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%] 2022-12-01T10:47:18.9804585Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%] 2022-12-01T10:47:18.9804977Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%] 2022-12-01T10:47:18.9805353Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED [ 57%] 2022-12-01T10:47:18.9805657Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED [ 58%] 2022-12-01T10:47:18.9806087Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%] 2022-12-01T10:47:18.9806423Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%] 2022-12-01T10:47:18.9806826Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%] 2022-12-01T10:47:18.9807162Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%] 2022-12-01T10:47:18.9807570Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%] 2022-12-01T10:47:18.9807932Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%] 2022-12-01T10:47:18.9808309Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED [ 71%] 2022-12-01T10:47:18.9808625Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED [ 73%] 2022-12-01T10:47:18.9808955Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions SKIPPED [ 75%] 2022-12-01T10:47:18.9809311Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED [ 76%] 2022-12-01T10:47:18.9809607Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED [ 78%] 2022-12-01T10:47:18.9809968Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED [ 80%] 2022-12-01T10:47:18.9810300Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%] 2022-12-01T10:47:18.9810680Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%] 2022-12-01T10:47:18.9811038Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device SKIPPED [ 85%] 2022-12-01T10:47:18.9811436Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules SKIPPED [ 87%] 2022-12-01T10:47:18.9811805Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%] 2022-12-01T10:47:18.9812193Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED [ 91%] 2022-12-01T10:47:18.9812524Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] SKIPPED [ 92%] 2022-12-01T10:47:18.9812918Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] SKIPPED [ 94%] 2022-12-01T10:47:18.9813253Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] SKIPPED [ 96%] 2022-12-01T10:47:18.9813640Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device SKIPPED [ 98%] 2022-12-01T10:47:18.9813968Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper SKIPPED [100%] 2022-12-01T10:47:18.9814146Z 2022-12-01T10:47:18.9814275Z ======================== 48 passed, 8 skipped in 1.54s ========================= 2022-12-01T10:47:18.9814459Z 2022-12-01T10:47:18.9814714Z ##[endgroup] 2022-12-01T10:47:18.9815295Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_0sa393lj) 2022-12-01T10:47:18.9815574Z 2022-12-01T10:47:18.9815817Z Running distributed/pipeline/sync/test_stream ... [2022-12-01 10:47:18.978467] 2022-12-01T10:47:18.9816345Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-12-01 10:47:18.978841] 2022-12-01T10:47:20.3306666Z 2022-12-01T10:47:20.3307231Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream 2022-12-01T10:47:20.3308429Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_mmpvedno) 2022-12-01T10:47:20.3309081Z ============================= test session starts ============================== 2022-12-01T10:47:20.3309721Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:20.3309982Z cachedir: .pytest_cache 2022-12-01T10:47:20.3310414Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:20.3310740Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:20.3310967Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:20.3311564Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:20.3311854Z collecting ... collected 19 items 2022-12-01T10:47:20.3313836Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view 2022-12-01T10:47:20.3315964Z 2022-12-01T10:47:20.3316143Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [ 5%] 2022-12-01T10:47:20.3316493Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda SKIPPED [ 10%] 2022-12-01T10:47:20.3316923Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%] 2022-12-01T10:47:20.3317310Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda SKIPPED [ 21%] 2022-12-01T10:47:20.3317729Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%] 2022-12-01T10:47:20.3318151Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda SKIPPED [ 31%] 2022-12-01T10:47:20.3318686Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%] 2022-12-01T10:47:20.3319094Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda SKIPPED [ 42%] 2022-12-01T10:47:20.3319448Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%] 2022-12-01T10:47:20.3319919Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda SKIPPED [ 52%] 2022-12-01T10:47:20.3320278Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%] 2022-12-01T10:47:20.3320683Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda SKIPPED [ 63%] 2022-12-01T10:47:20.3321028Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%] 2022-12-01T10:47:20.3321450Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda SKIPPED [ 73%] 2022-12-01T10:47:20.3321874Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu SKIPPED [ 78%] 2022-12-01T10:47:20.3322285Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda SKIPPED [ 84%] 2022-12-01T10:47:20.3322653Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%] 2022-12-01T10:47:20.3323064Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda SKIPPED [ 94%] 2022-12-01T10:47:20.3323487Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view SKIPPED [100%] 2022-12-01T10:47:20.3323704Z 2022-12-01T10:47:20.3323823Z ======================== 8 passed, 11 skipped in 0.05s ========================= 2022-12-01T10:47:20.3323956Z 2022-12-01T10:47:20.3324238Z ##[endgroup] 2022-12-01T10:47:20.3324754Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_mmpvedno) 2022-12-01T10:47:20.3325087Z 2022-12-01T10:47:20.3325303Z Running distributed/pipeline/sync/test_worker ... [2022-12-01 10:47:20.331033] 2022-12-01T10:47:20.3325842Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-12-01 10:47:20.331362] 2022-12-01T10:47:21.6394031Z 2022-12-01T10:47:21.6394645Z Expand the folded group to see the log file of distributed/pipeline/sync/test_worker 2022-12-01T10:47:21.6395915Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_x48dlvna) 2022-12-01T10:47:21.6396346Z ============================= test session starts ============================== 2022-12-01T10:47:21.6396827Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-12-01T10:47:21.6397089Z cachedir: .pytest_cache 2022-12-01T10:47:21.6397520Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-12-01T10:47:21.6397844Z torch: 1.13.0a0+gitc13d400 2022-12-01T10:47:21.6398084Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-12-01T10:47:21.6398670Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2 2022-12-01T10:47:21.6398948Z collecting ... collected 6 items 2022-12-01T10:47:21.6399614Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device 2022-12-01T10:47:21.6400145Z 2022-12-01T10:47:21.6400312Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%] 2022-12-01T10:47:21.6400649Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED [ 33%] 2022-12-01T10:47:21.6400974Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED [ 50%] 2022-12-01T10:47:21.6401282Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED [ 66%] 2022-12-01T10:47:21.6401597Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED [ 83%] 2022-12-01T10:47:21.6402094Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED [100%] 2022-12-01T10:47:21.6402278Z 2022-12-01T10:47:21.6402376Z ============================== 6 passed in 0.03s =============================== 2022-12-01T10:47:21.6402514Z 2022-12-01T10:47:21.6402755Z ##[endgroup] 2022-12-01T10:47:21.6403243Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_x48dlvna) 2022-12-01T10:47:21.6403587Z 2022-12-01T10:47:21.8882812Z 2022-12-01T10:47:21.8883159Z real 42m40.538s 2022-12-01T10:47:21.8883538Z user 86m56.042s 2022-12-01T10:47:21.8883835Z sys 28m3.862s 2022-12-01T10:47:21.8884101Z + assert_git_not_dirty 2022-12-01T10:47:21.8884550Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]] 2022-12-01T10:47:21.8884826Z + [[ linux-focal-py3.7-gcc7 != *xla* ]] 2022-12-01T10:47:21.8885717Z ++ git status --porcelain 2022-12-01T10:47:22.7867587Z + git_status= 2022-12-01T10:47:22.7868164Z + [[ -n '' ]] 2022-12-01T10:47:22.7868617Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]] 2022-12-01T10:47:22.7868884Z + [[ 1 == 1 ]] 2022-12-01T10:47:22.7869052Z + test_rpc 2022-12-01T10:47:22.7869285Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]] 2022-12-01T10:47:22.7869596Z + echo 'Testing RPC C++ tests' 2022-12-01T10:47:22.7869806Z Testing RPC C++ tests 2022-12-01T10:47:22.7870669Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-12-01T10:47:22.7880039Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-12-01T10:47:22.7888382Z + ln -sf '/opt/conda/lib/python3.7/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.7/site-packages/torch/bin 2022-12-01T10:47:22.7895715Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc 2022-12-01T10:47:22.7896031Z + mkdir -p test/test-reports/cpp-rpc/test_rpc 2022-12-01T10:47:22.7907181Z + /opt/conda/lib/python3.7/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml 2022-12-01T10:47:23.0409935Z CUDA not available. Disabling CUDA and MultiCUDA tests 2022-12-01T10:47:23.0410751Z Note: Google Test filter = *-*_CUDA:*_MultiCUDA 2022-12-01T10:47:23.0411085Z [==========] Running 8 tests from 3 test suites. 2022-12-01T10:47:23.0411398Z [----------] Global test environment set-up. 2022-12-01T10:47:23.0411684Z [----------] 4 tests from WireSerialize 2022-12-01T10:47:23.0411958Z [ RUN ] WireSerialize.Base 2022-12-01T10:47:23.0606663Z [ OK ] WireSerialize.Base (19 ms) 2022-12-01T10:47:23.0607013Z [ RUN ] WireSerialize.RecopySparseTensors 2022-12-01T10:47:23.0684734Z [ OK ] WireSerialize.RecopySparseTensors (7 ms) 2022-12-01T10:47:23.0685090Z [ RUN ] WireSerialize.CloneSparseTensors 2022-12-01T10:47:23.0757745Z [ OK ] WireSerialize.CloneSparseTensors (7 ms) 2022-12-01T10:47:23.0758048Z [ RUN ] WireSerialize.Errors 2022-12-01T10:47:23.0781987Z [ OK ] WireSerialize.Errors (2 ms) 2022-12-01T10:47:23.0782561Z [----------] 4 tests from WireSerialize (37 ms total) 2022-12-01T10:47:23.0782866Z 2022-12-01T10:47:23.0783033Z [----------] 1 test from TestE2ETensorPipe 2022-12-01T10:47:23.0783354Z [ RUN ] TestE2ETensorPipe.TestTrainingLoop 2022-12-01T10:47:23.4410423Z [W tensorpipe_agent.cpp:726] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-12-01T10:47:23.4420043Z [ OK ] TestE2ETensorPipe.TestTrainingLoop (363 ms) 2022-12-01T10:47:23.4421000Z [----------] 1 test from TestE2ETensorPipe (363 ms total) 2022-12-01T10:47:23.4421338Z 2022-12-01T10:47:23.4421673Z [----------] 3 tests from TensorpipeSerialize 2022-12-01T10:47:23.4421989Z [ RUN ] TensorpipeSerialize.Base 2022-12-01T10:47:23.4422281Z [ OK ] TensorpipeSerialize.Base (0 ms) 2022-12-01T10:47:23.4422626Z [ RUN ] TensorpipeSerialize.RecopySparseTensors 2022-12-01T10:47:23.4489721Z [ OK ] TensorpipeSerialize.RecopySparseTensors (6 ms) 2022-12-01T10:47:23.4490214Z [ RUN ] TensorpipeSerialize.NoDeleterTensors 2022-12-01T10:47:23.4490697Z [ OK ] TensorpipeSerialize.NoDeleterTensors (0 ms) 2022-12-01T10:47:23.4491191Z [----------] 3 tests from TensorpipeSerialize (6 ms total) 2022-12-01T10:47:23.4491413Z 2022-12-01T10:47:23.4491638Z [----------] Global test environment tear-down 2022-12-01T10:47:23.4493450Z [==========] 8 tests from 3 test suites ran. (408 ms total) 2022-12-01T10:47:23.4493809Z [ PASSED ] 8 tests. 2022-12-01T10:47:23.4493964Z 2022-12-01T10:47:23.4494128Z  YOU HAVE 1 DISABLED TEST 2022-12-01T10:47:23.4494286Z 2022-12-01T10:47:23.5175914Z  2022-12-01T10:47:23.5206384Z Prepare all required actions 2022-12-01T10:47:23.5206694Z Getting action download info 2022-12-01T10:47:23.9825675Z ##[group]Run ./.github/actions/get-workflow-job-id 2022-12-01T10:47:23.9826062Z with: 2022-12-01T10:47:23.9826544Z github-token: *** 2022-12-01T10:47:23.9826707Z env: 2022-12-01T10:47:23.9826879Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:23.9827066Z ##[endgroup] 2022-12-01T10:47:23.9853829Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767 2022-12-01T10:47:23.9854069Z with: 2022-12-01T10:47:23.9854217Z shell: bash 2022-12-01T10:47:23.9854393Z timeout_minutes: 10 2022-12-01T10:47:23.9854573Z max_attempts: 5 2022-12-01T10:47:23.9854744Z retry_wait_seconds: 30 2022-12-01T10:47:23.9855128Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}" 2022-12-01T10:47:23.9855558Z polling_interval_seconds: 1 2022-12-01T10:47:23.9855742Z warning_on_retry: true 2022-12-01T10:47:23.9855931Z continue_on_error: false 2022-12-01T10:47:23.9856108Z env: 2022-12-01T10:47:23.9856265Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:23.9856591Z GITHUB_TOKEN: *** 2022-12-01T10:47:23.9856769Z ##[endgroup] 2022-12-01T10:47:24.3320016Z 2022-12-01T10:47:24.3325573Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-12-01T10:47:24.3362841Z + python3 -m pip install requests==2.26.0 2022-12-01T10:47:24.5434045Z Defaulting to user installation because normal site-packages is not writeable 2022-12-01T10:47:24.5614828Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0) 2022-12-01T10:47:24.5741725Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24) 2022-12-01T10:47:24.5750146Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.13) 2022-12-01T10:47:24.5901448Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12) 2022-12-01T10:47:24.5921612Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4) 2022-12-01T10:47:24.8750369Z ++ python3 .github/scripts/get_workflow_job_id.py 3591403534 i-092913c5169eb9727 2022-12-01T10:47:27.4538887Z + GHA_WORKFLOW_JOB_ID=9818325812 2022-12-01T10:47:27.4539420Z + echo '::set-output name=job-id::9818325812' 2022-12-01T10:47:27.4547111Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-12-01T10:47:28.3391797Z Command completed after 1 attempt(s). 2022-12-01T10:47:28.3392212Z 2022-12-01T10:47:28.3394054Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-12-01T10:47:28.3495659Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2022-12-01T10:47:28.3495904Z kill "$MONITOR_SCRIPT_PID" 2022-12-01T10:47:28.3507266Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:47:28.3507490Z env: 2022-12-01T10:47:28.3507654Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:28.3507854Z MONITOR_SCRIPT_PID: 11499 2022-12-01T10:47:28.3508036Z ##[endgroup] 2022-12-01T10:47:28.3581643Z Prepare all required actions 2022-12-01T10:47:28.3581892Z Getting action download info 2022-12-01T10:47:28.5209269Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2) 2022-12-01T10:47:28.6607004Z ##[group]Run ./.github/actions/upload-test-artifacts 2022-12-01T10:47:28.6607220Z with: 2022-12-01T10:47:28.6607453Z file-suffix: test-distributed-1-2-linux.2xlarge_9818325812 2022-12-01T10:47:28.6607681Z env: 2022-12-01T10:47:28.6607838Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:28.6608021Z ##[endgroup] 2022-12-01T10:47:28.6631389Z ##[group]Run # Remove any previous test jsons if they exist 2022-12-01T10:47:28.6631676Z # Remove any previous test jsons if they exist 2022-12-01T10:47:28.6631908Z rm -f test-jsons-*.zip 2022-12-01T10:47:28.6632154Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2022-12-01T10:47:28.6643483Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:47:28.6643734Z env: 2022-12-01T10:47:28.6643916Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:28.6644158Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_9818325812 2022-12-01T10:47:28.6644394Z ##[endgroup] 2022-12-01T10:47:28.6741976Z adding: test/allowlist_for_publicAPI.json (deflated 80%) 2022-12-01T10:47:28.6768711Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2022-12-01T10:47:28.6774548Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2022-12-01T10:47:28.6775799Z adding: test/.pytorch-slow-tests.json (deflated 76%) 2022-12-01T10:47:28.6784141Z adding: test/.pytorch-disabled-tests.json (deflated 86%) 2022-12-01T10:47:28.6803774Z ##[group]Run # Remove any previous test reports if they exist 2022-12-01T10:47:28.6804070Z # Remove any previous test reports if they exist 2022-12-01T10:47:28.6804308Z rm -f test-reports-*.zip 2022-12-01T10:47:28.6804557Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' 2022-12-01T10:47:28.6815158Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:47:28.6815372Z env: 2022-12-01T10:47:28.6815547Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:28.6815786Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_9818325812 2022-12-01T10:47:28.6816021Z ##[endgroup] 2022-12-01T10:47:28.6936259Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20221201100445.xml (deflated 81%) 2022-12-01T10:47:28.6936844Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20221201100445.xml (deflated 64%) 2022-12-01T10:47:28.6937398Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20221201100445.xml (deflated 62%) 2022-12-01T10:47:28.6938503Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20221201100445.xml (deflated 93%) 2022-12-01T10:47:28.6940559Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221201100559.xml (deflated 94%) 2022-12-01T10:47:28.6941330Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221201100559.xml (deflated 87%) 2022-12-01T10:47:28.6941919Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100730.xml (deflated 40%) 2022-12-01T10:47:28.6942561Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100731.xml (deflated 41%) 2022-12-01T10:47:28.6943077Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100732.xml (deflated 40%) 2022-12-01T10:47:28.6943572Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100734.xml (deflated 40%) 2022-12-01T10:47:28.6944071Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100735.xml (deflated 39%) 2022-12-01T10:47:28.6944632Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100737.xml (deflated 40%) 2022-12-01T10:47:28.6945144Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100738.xml (deflated 40%) 2022-12-01T10:47:28.6945625Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100739.xml (deflated 40%) 2022-12-01T10:47:28.6946123Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100741.xml (deflated 40%) 2022-12-01T10:47:28.6946623Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100742.xml (deflated 40%) 2022-12-01T10:47:28.6947117Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100743.xml (deflated 39%) 2022-12-01T10:47:28.6947602Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100745.xml (deflated 40%) 2022-12-01T10:47:28.6948105Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100746.xml (deflated 40%) 2022-12-01T10:47:28.6948595Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100748.xml (deflated 39%) 2022-12-01T10:47:28.6949087Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100749.xml (deflated 39%) 2022-12-01T10:47:28.6949569Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100750.xml (deflated 40%) 2022-12-01T10:47:28.6950064Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100752.xml (deflated 40%) 2022-12-01T10:47:28.6950555Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100753.xml (deflated 40%) 2022-12-01T10:47:28.6951049Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221201100755.xml (deflated 40%) 2022-12-01T10:47:28.6951548Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100756.xml (deflated 40%) 2022-12-01T10:47:28.6952080Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100759.xml (deflated 40%) 2022-12-01T10:47:28.6952604Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100802.xml (deflated 40%) 2022-12-01T10:47:28.6953119Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100806.xml (deflated 40%) 2022-12-01T10:47:28.6953617Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100809.xml (deflated 40%) 2022-12-01T10:47:28.6954147Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100812.xml (deflated 40%) 2022-12-01T10:47:28.6954669Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221201100815.xml (deflated 40%) 2022-12-01T10:47:28.6955243Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100819.xml (deflated 42%) 2022-12-01T10:47:28.6955847Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100820.xml (deflated 42%) 2022-12-01T10:47:28.6956495Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100821.xml (deflated 42%) 2022-12-01T10:47:28.6957108Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100823.xml (deflated 42%) 2022-12-01T10:47:28.6957716Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100824.xml (deflated 42%) 2022-12-01T10:47:28.6958537Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100825.xml (deflated 43%) 2022-12-01T10:47:28.6959180Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100827.xml (deflated 42%) 2022-12-01T10:47:28.6959784Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100828.xml (deflated 42%) 2022-12-01T10:47:28.6960389Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100830.xml (deflated 43%) 2022-12-01T10:47:28.6960999Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100831.xml (deflated 45%) 2022-12-01T10:47:28.6961585Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100834.xml (deflated 45%) 2022-12-01T10:47:28.6962179Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100838.xml (deflated 43%) 2022-12-01T10:47:28.6962778Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100841.xml (deflated 43%) 2022-12-01T10:47:28.6963383Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100844.xml (deflated 45%) 2022-12-01T10:47:28.6963972Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100847.xml (deflated 45%) 2022-12-01T10:47:28.6964576Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100851.xml (deflated 47%) 2022-12-01T10:47:28.6965179Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100854.xml (deflated 46%) 2022-12-01T10:47:28.6965833Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100857.xml (deflated 45%) 2022-12-01T10:47:28.6966430Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100900.xml (deflated 45%) 2022-12-01T10:47:28.6967028Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100903.xml (deflated 45%) 2022-12-01T10:47:28.6967635Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100907.xml (deflated 44%) 2022-12-01T10:47:28.6968232Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100910.xml (deflated 44%) 2022-12-01T10:47:28.6968817Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100913.xml (deflated 43%) 2022-12-01T10:47:28.6969417Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100915.xml (deflated 42%) 2022-12-01T10:47:28.6970036Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100916.xml (deflated 42%) 2022-12-01T10:47:28.6970631Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100917.xml (deflated 45%) 2022-12-01T10:47:28.6971289Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100919.xml (deflated 45%) 2022-12-01T10:47:28.6971875Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100920.xml (deflated 42%) 2022-12-01T10:47:28.6972473Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100922.xml (deflated 42%) 2022-12-01T10:47:28.6973071Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100923.xml (deflated 42%) 2022-12-01T10:47:28.6973707Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100924.xml (deflated 43%) 2022-12-01T10:47:28.6974296Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100926.xml (deflated 43%) 2022-12-01T10:47:28.6974893Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100927.xml (deflated 43%) 2022-12-01T10:47:28.6975495Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100928.xml (deflated 42%) 2022-12-01T10:47:28.6976089Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100930.xml (deflated 42%) 2022-12-01T10:47:28.6976680Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100931.xml (deflated 42%) 2022-12-01T10:47:28.6977282Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100933.xml (deflated 42%) 2022-12-01T10:47:28.6977879Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100934.xml (deflated 41%) 2022-12-01T10:47:28.6978477Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100935.xml (deflated 41%) 2022-12-01T10:47:28.6979061Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100937.xml (deflated 43%) 2022-12-01T10:47:28.6979654Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100938.xml (deflated 42%) 2022-12-01T10:47:28.6980254Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100940.xml (deflated 43%) 2022-12-01T10:47:28.6980850Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100941.xml (deflated 43%) 2022-12-01T10:47:28.6981436Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100942.xml (deflated 42%) 2022-12-01T10:47:28.6982031Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100944.xml (deflated 42%) 2022-12-01T10:47:28.6982631Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100945.xml (deflated 41%) 2022-12-01T10:47:28.6983231Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100948.xml (deflated 43%) 2022-12-01T10:47:28.6983811Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100950.xml (deflated 43%) 2022-12-01T10:47:28.6984410Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100951.xml (deflated 42%) 2022-12-01T10:47:28.6985009Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100952.xml (deflated 43%) 2022-12-01T10:47:28.6985607Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100954.xml (deflated 43%) 2022-12-01T10:47:28.6986227Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100955.xml (deflated 43%) 2022-12-01T10:47:28.6986825Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100957.xml (deflated 43%) 2022-12-01T10:47:28.6987421Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100958.xml (deflated 43%) 2022-12-01T10:47:28.6988052Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201100959.xml (deflated 43%) 2022-12-01T10:47:28.6988691Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101001.xml (deflated 44%) 2022-12-01T10:47:28.6989281Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101002.xml (deflated 43%) 2022-12-01T10:47:28.6989876Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101004.xml (deflated 45%) 2022-12-01T10:47:28.6990475Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101005.xml (deflated 43%) 2022-12-01T10:47:28.6991073Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101006.xml (deflated 43%) 2022-12-01T10:47:28.6991658Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101008.xml (deflated 42%) 2022-12-01T10:47:28.6992257Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101009.xml (deflated 42%) 2022-12-01T10:47:28.6992852Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101010.xml (deflated 43%) 2022-12-01T10:47:28.6993457Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221201101014.xml (deflated 42%) 2022-12-01T10:47:28.6994026Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101017.xml (deflated 42%) 2022-12-01T10:47:28.6994602Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101018.xml (deflated 42%) 2022-12-01T10:47:28.6995175Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101020.xml (deflated 42%) 2022-12-01T10:47:28.6995753Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101021.xml (deflated 42%) 2022-12-01T10:47:28.6996306Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101022.xml (deflated 42%) 2022-12-01T10:47:28.6996874Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101024.xml (deflated 42%) 2022-12-01T10:47:28.6997446Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101025.xml (deflated 42%) 2022-12-01T10:47:28.6998017Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101027.xml (deflated 42%) 2022-12-01T10:47:28.6998701Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221201101028.xml (deflated 43%) 2022-12-01T10:47:28.6999385Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221201101029.xml (deflated 45%) 2022-12-01T10:47:28.7000065Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221201101031.xml (deflated 42%) 2022-12-01T10:47:28.7000653Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101032.xml (deflated 42%) 2022-12-01T10:47:28.7001270Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101034.xml (deflated 42%) 2022-12-01T10:47:28.7001841Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101035.xml (deflated 42%) 2022-12-01T10:47:28.7002403Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101036.xml (deflated 42%) 2022-12-01T10:47:28.7003009Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101038.xml (deflated 42%) 2022-12-01T10:47:28.7003567Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101039.xml (deflated 42%) 2022-12-01T10:47:28.7004129Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101040.xml (deflated 42%) 2022-12-01T10:47:28.7004697Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101042.xml (deflated 42%) 2022-12-01T10:47:28.7005264Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101043.xml (deflated 42%) 2022-12-01T10:47:28.7005867Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101045.xml (deflated 42%) 2022-12-01T10:47:28.7006440Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101046.xml (deflated 42%) 2022-12-01T10:47:28.7007009Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101047.xml (deflated 41%) 2022-12-01T10:47:28.7007573Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101049.xml (deflated 42%) 2022-12-01T10:47:28.7008135Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101050.xml (deflated 42%) 2022-12-01T10:47:28.7008688Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101051.xml (deflated 42%) 2022-12-01T10:47:28.7009253Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101053.xml (deflated 42%) 2022-12-01T10:47:28.7009816Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101054.xml (deflated 42%) 2022-12-01T10:47:28.7010375Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221201101056.xml (deflated 42%) 2022-12-01T10:47:28.7010916Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221201101057.xml (deflated 41%) 2022-12-01T10:47:28.7011450Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221201101058.xml (deflated 41%) 2022-12-01T10:47:28.7012014Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221201101100.xml (deflated 81%) 2022-12-01T10:47:28.7012587Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221201101111.xml (deflated 81%) 2022-12-01T10:47:28.7013106Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221201101132.xml (deflated 41%) 2022-12-01T10:47:28.7013620Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221201101134.xml (deflated 40%) 2022-12-01T10:47:28.7014143Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221201101135.xml (deflated 40%) 2022-12-01T10:47:28.7014650Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221201101136.xml (deflated 40%) 2022-12-01T10:47:28.7015189Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221201101138.xml (deflated 41%) 2022-12-01T10:47:28.7015770Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221201101139.xml (deflated 41%) 2022-12-01T10:47:28.7016312Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221201101140.xml (deflated 41%) 2022-12-01T10:47:28.7016842Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221201101141.xml (deflated 40%) 2022-12-01T10:47:28.7017424Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221201101143.xml (deflated 40%) 2022-12-01T10:47:28.7017966Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221201101144.xml (deflated 40%) 2022-12-01T10:47:28.7018502Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221201101145.xml (deflated 40%) 2022-12-01T10:47:28.7019096Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221201101147.xml (deflated 40%) 2022-12-01T10:47:28.7019627Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221201101148.xml (deflated 41%) 2022-12-01T10:47:28.7020168Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201101149.xml (deflated 40%) 2022-12-01T10:47:28.7020709Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201101151.xml (deflated 40%) 2022-12-01T10:47:28.7021246Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201101152.xml (deflated 40%) 2022-12-01T10:47:28.7021763Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201101153.xml (deflated 40%) 2022-12-01T10:47:28.7022286Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221201101205.xml (deflated 40%) 2022-12-01T10:47:28.7022813Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221201101206.xml (deflated 40%) 2022-12-01T10:47:28.7023326Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101207.xml (deflated 39%) 2022-12-01T10:47:28.7023834Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101209.xml (deflated 40%) 2022-12-01T10:47:28.7024348Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101210.xml (deflated 39%) 2022-12-01T10:47:28.7024862Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101212.xml (deflated 39%) 2022-12-01T10:47:28.7025374Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101213.xml (deflated 39%) 2022-12-01T10:47:28.7025870Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101214.xml (deflated 40%) 2022-12-01T10:47:28.7026386Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101216.xml (deflated 40%) 2022-12-01T10:47:28.7026894Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201101219.xml (deflated 40%) 2022-12-01T10:47:28.7027467Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221201101220.xml (deflated 87%) 2022-12-01T10:47:28.7028057Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221201101233.xml (deflated 84%) 2022-12-01T10:47:28.7028654Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221201101245.xml (deflated 87%) 2022-12-01T10:47:28.7029278Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221201101255.xml (deflated 77%) 2022-12-01T10:47:28.7029889Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221201101303.xml (deflated 64%) 2022-12-01T10:47:28.7030430Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221201101308.xml (deflated 59%) 2022-12-01T10:47:28.7030972Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221201101312.xml (deflated 59%) 2022-12-01T10:47:28.7031518Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221201101316.xml (deflated 57%) 2022-12-01T10:47:28.7032156Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221201101319.xml (deflated 47%) 2022-12-01T10:47:28.7032793Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221201101321.xml (deflated 67%) 2022-12-01T10:47:28.7033430Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221201101321.xml (deflated 78%) 2022-12-01T10:47:28.7034010Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221201101324.xml (deflated 43%) 2022-12-01T10:47:28.7034580Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221201101327.xml (deflated 45%) 2022-12-01T10:47:28.7035205Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221201101330.xml (deflated 78%) 2022-12-01T10:47:28.7035860Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221201101330.xml (deflated 61%) 2022-12-01T10:47:28.7036563Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221201101334.xml (deflated 69%) 2022-12-01T10:47:28.7037298Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221201101334.xml (deflated 43%) 2022-12-01T10:47:28.7038113Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221201101334.xml (deflated 45%) 2022-12-01T10:47:28.7038956Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_utils/TEST-TestMedatadaIndex-20221201101342.xml (deflated 72%) 2022-12-01T10:47:28.7039570Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20221201101343.xml (deflated 81%) 2022-12-01T10:47:28.7040191Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20221201101343.xml (deflated 85%) 2022-12-01T10:47:28.7040838Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20221201101343.xml (deflated 85%) 2022-12-01T10:47:28.7041501Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221201101346.xml (deflated 42%) 2022-12-01T10:47:28.7042141Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221201101348.xml (deflated 42%) 2022-12-01T10:47:28.7042791Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221201101349.xml (deflated 41%) 2022-12-01T10:47:28.7043441Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221201101353.xml (deflated 42%) 2022-12-01T10:47:28.7044127Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221201101357.xml (deflated 42%) 2022-12-01T10:47:28.7044875Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221201101401.xml (deflated 42%) 2022-12-01T10:47:28.7045581Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221201101406.xml (deflated 42%) 2022-12-01T10:47:28.7046336Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221201101411.xml (deflated 42%) 2022-12-01T10:47:28.7047048Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101415.xml (deflated 44%) 2022-12-01T10:47:28.7047684Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101419.xml (deflated 42%) 2022-12-01T10:47:28.7048325Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101423.xml (deflated 41%) 2022-12-01T10:47:28.7048975Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101426.xml (deflated 42%) 2022-12-01T10:47:28.7049617Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101430.xml (deflated 41%) 2022-12-01T10:47:28.7050245Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101434.xml (deflated 41%) 2022-12-01T10:47:28.7050895Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101437.xml (deflated 41%) 2022-12-01T10:47:28.7051533Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101441.xml (deflated 41%) 2022-12-01T10:47:28.7052176Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101445.xml (deflated 42%) 2022-12-01T10:47:28.7052814Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101446.xml (deflated 40%) 2022-12-01T10:47:28.7053439Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101450.xml (deflated 42%) 2022-12-01T10:47:28.7054067Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101451.xml (deflated 41%) 2022-12-01T10:47:28.7054704Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101455.xml (deflated 41%) 2022-12-01T10:47:28.7055336Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101458.xml (deflated 41%) 2022-12-01T10:47:28.7055959Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101502.xml (deflated 41%) 2022-12-01T10:47:28.7056602Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101506.xml (deflated 41%) 2022-12-01T10:47:28.7057235Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101509.xml (deflated 41%) 2022-12-01T10:47:28.7057870Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101513.xml (deflated 41%) 2022-12-01T10:47:28.7058489Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101516.xml (deflated 41%) 2022-12-01T10:47:28.7059127Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101520.xml (deflated 41%) 2022-12-01T10:47:28.7059790Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101523.xml (deflated 41%) 2022-12-01T10:47:28.7060422Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101527.xml (deflated 41%) 2022-12-01T10:47:28.7061046Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101531.xml (deflated 41%) 2022-12-01T10:47:28.7061713Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101544.xml (deflated 42%) 2022-12-01T10:47:28.7062351Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101546.xml (deflated 40%) 2022-12-01T10:47:28.7062983Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101550.xml (deflated 41%) 2022-12-01T10:47:28.7063622Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101553.xml (deflated 40%) 2022-12-01T10:47:28.7064236Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101557.xml (deflated 41%) 2022-12-01T10:47:28.7064876Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101600.xml (deflated 49%) 2022-12-01T10:47:28.7065518Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101604.xml (deflated 41%) 2022-12-01T10:47:28.7066158Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101608.xml (deflated 42%) 2022-12-01T10:47:28.7066782Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101611.xml (deflated 42%) 2022-12-01T10:47:28.7067421Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101615.xml (deflated 41%) 2022-12-01T10:47:28.7068056Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101619.xml (deflated 41%) 2022-12-01T10:47:28.7068698Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101623.xml (deflated 42%) 2022-12-01T10:47:28.7069414Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101624.xml (deflated 41%) 2022-12-01T10:47:28.7070049Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101628.xml (deflated 41%) 2022-12-01T10:47:28.7070673Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101631.xml (deflated 41%) 2022-12-01T10:47:28.7071315Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101635.xml (deflated 41%) 2022-12-01T10:47:28.7071955Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101639.xml (deflated 41%) 2022-12-01T10:47:28.7072587Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101643.xml (deflated 41%) 2022-12-01T10:47:28.7073226Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101646.xml (deflated 41%) 2022-12-01T10:47:28.7073846Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101650.xml (deflated 41%) 2022-12-01T10:47:28.7074520Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101654.xml (deflated 41%) 2022-12-01T10:47:28.7075151Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101657.xml (deflated 41%) 2022-12-01T10:47:28.7075786Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101701.xml (deflated 41%) 2022-12-01T10:47:28.7076434Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101705.xml (deflated 41%) 2022-12-01T10:47:28.7077073Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101709.xml (deflated 41%) 2022-12-01T10:47:28.7077715Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101712.xml (deflated 41%) 2022-12-01T10:47:28.7078467Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101717.xml (deflated 41%) 2022-12-01T10:47:28.7079101Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101720.xml (deflated 41%) 2022-12-01T10:47:28.7079737Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101724.xml (deflated 41%) 2022-12-01T10:47:28.7080375Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101727.xml (deflated 44%) 2022-12-01T10:47:28.7081017Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101731.xml (deflated 41%) 2022-12-01T10:47:28.7081636Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101735.xml (deflated 42%) 2022-12-01T10:47:28.7082279Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101738.xml (deflated 41%) 2022-12-01T10:47:28.7082914Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101742.xml (deflated 41%) 2022-12-01T10:47:28.7083556Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101745.xml (deflated 41%) 2022-12-01T10:47:28.7084187Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101749.xml (deflated 41%) 2022-12-01T10:47:28.7084808Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101753.xml (deflated 41%) 2022-12-01T10:47:28.7085446Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101757.xml (deflated 41%) 2022-12-01T10:47:28.7086144Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101801.xml (deflated 41%) 2022-12-01T10:47:28.7086789Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221201101805.xml (deflated 41%) 2022-12-01T10:47:28.7087417Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221201101808.xml (deflated 43%) 2022-12-01T10:47:28.7088069Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221201101810.xml (deflated 42%) 2022-12-01T10:47:28.7088712Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221201101813.xml (deflated 42%) 2022-12-01T10:47:28.7089413Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221201101817.xml (deflated 42%) 2022-12-01T10:47:28.7090059Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221201101819.xml (deflated 41%) 2022-12-01T10:47:28.7090724Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221201101822.xml (deflated 42%) 2022-12-01T10:47:28.7091394Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221201101826.xml (deflated 41%) 2022-12-01T10:47:28.7092096Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221201101829.xml (deflated 41%) 2022-12-01T10:47:28.7092720Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101833.xml (deflated 40%) 2022-12-01T10:47:28.7093325Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101837.xml (deflated 39%) 2022-12-01T10:47:28.7093931Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101840.xml (deflated 40%) 2022-12-01T10:47:28.7094535Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101844.xml (deflated 41%) 2022-12-01T10:47:28.7095134Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101848.xml (deflated 40%) 2022-12-01T10:47:28.7095727Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101851.xml (deflated 40%) 2022-12-01T10:47:28.7096329Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101855.xml (deflated 40%) 2022-12-01T10:47:28.7096925Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101858.xml (deflated 40%) 2022-12-01T10:47:28.7097519Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101902.xml (deflated 40%) 2022-12-01T10:47:28.7098095Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101906.xml (deflated 40%) 2022-12-01T10:47:28.7098694Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101909.xml (deflated 41%) 2022-12-01T10:47:28.7099286Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101913.xml (deflated 41%) 2022-12-01T10:47:28.7099877Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101916.xml (deflated 40%) 2022-12-01T10:47:28.7100456Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101919.xml (deflated 40%) 2022-12-01T10:47:28.7101049Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101923.xml (deflated 40%) 2022-12-01T10:47:28.7101657Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101927.xml (deflated 40%) 2022-12-01T10:47:28.7102253Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101930.xml (deflated 40%) 2022-12-01T10:47:28.7102839Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101934.xml (deflated 40%) 2022-12-01T10:47:28.7103433Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101937.xml (deflated 40%) 2022-12-01T10:47:28.7104054Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101941.xml (deflated 40%) 2022-12-01T10:47:28.7104655Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101944.xml (deflated 41%) 2022-12-01T10:47:28.7105233Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101948.xml (deflated 40%) 2022-12-01T10:47:28.7105848Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101951.xml (deflated 41%) 2022-12-01T10:47:28.7106444Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101955.xml (deflated 41%) 2022-12-01T10:47:28.7107046Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201101957.xml (deflated 40%) 2022-12-01T10:47:28.7107635Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102000.xml (deflated 41%) 2022-12-01T10:47:28.7108227Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102004.xml (deflated 40%) 2022-12-01T10:47:28.7108819Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102007.xml (deflated 40%) 2022-12-01T10:47:28.7109413Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102011.xml (deflated 41%) 2022-12-01T10:47:28.7110002Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102015.xml (deflated 40%) 2022-12-01T10:47:28.7110585Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102018.xml (deflated 41%) 2022-12-01T10:47:28.7111173Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102022.xml (deflated 40%) 2022-12-01T10:47:28.7111763Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102026.xml (deflated 39%) 2022-12-01T10:47:28.7112347Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102030.xml (deflated 41%) 2022-12-01T10:47:28.7112926Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102033.xml (deflated 41%) 2022-12-01T10:47:28.7113518Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102037.xml (deflated 40%) 2022-12-01T10:47:28.7114103Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102040.xml (deflated 40%) 2022-12-01T10:47:28.7114694Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102044.xml (deflated 41%) 2022-12-01T10:47:28.7115268Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102048.xml (deflated 41%) 2022-12-01T10:47:28.7115858Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102051.xml (deflated 40%) 2022-12-01T10:47:28.7116458Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102055.xml (deflated 40%) 2022-12-01T10:47:28.7117049Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102058.xml (deflated 40%) 2022-12-01T10:47:28.7117630Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102102.xml (deflated 41%) 2022-12-01T10:47:28.7118389Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102106.xml (deflated 40%) 2022-12-01T10:47:28.7119009Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102110.xml (deflated 41%) 2022-12-01T10:47:28.7119607Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102113.xml (deflated 41%) 2022-12-01T10:47:28.7120239Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102117.xml (deflated 40%) 2022-12-01T10:47:28.7120836Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102120.xml (deflated 40%) 2022-12-01T10:47:28.7121427Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102124.xml (deflated 40%) 2022-12-01T10:47:28.7122021Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102128.xml (deflated 41%) 2022-12-01T10:47:28.7122599Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102131.xml (deflated 40%) 2022-12-01T10:47:28.7123193Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102135.xml (deflated 40%) 2022-12-01T10:47:28.7123785Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102139.xml (deflated 41%) 2022-12-01T10:47:28.7124376Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221201102142.xml (deflated 41%) 2022-12-01T10:47:28.7125016Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20221201102146.xml (deflated 43%) 2022-12-01T10:47:28.7125762Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20221201102149.xml (deflated 43%) 2022-12-01T10:47:28.7126464Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102153.xml (deflated 42%) 2022-12-01T10:47:28.7127109Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102156.xml (deflated 41%) 2022-12-01T10:47:28.7127759Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102200.xml (deflated 41%) 2022-12-01T10:47:28.7128389Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102203.xml (deflated 41%) 2022-12-01T10:47:28.7129038Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102207.xml (deflated 41%) 2022-12-01T10:47:28.7129681Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102210.xml (deflated 41%) 2022-12-01T10:47:28.7130322Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102213.xml (deflated 42%) 2022-12-01T10:47:28.7130943Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102217.xml (deflated 42%) 2022-12-01T10:47:28.7131584Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102220.xml (deflated 42%) 2022-12-01T10:47:28.7132224Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102223.xml (deflated 42%) 2022-12-01T10:47:28.7132864Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102226.xml (deflated 42%) 2022-12-01T10:47:28.7133533Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102230.xml (deflated 42%) 2022-12-01T10:47:28.7134174Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221201102233.xml (deflated 41%) 2022-12-01T10:47:28.7134785Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102236.xml (deflated 41%) 2022-12-01T10:47:28.7135420Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102240.xml (deflated 41%) 2022-12-01T10:47:28.7136008Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102243.xml (deflated 40%) 2022-12-01T10:47:28.7136583Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102247.xml (deflated 40%) 2022-12-01T10:47:28.7137175Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102250.xml (deflated 41%) 2022-12-01T10:47:28.7137766Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102303.xml (deflated 41%) 2022-12-01T10:47:28.7138352Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102306.xml (deflated 40%) 2022-12-01T10:47:28.7138924Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102310.xml (deflated 41%) 2022-12-01T10:47:28.7139512Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102313.xml (deflated 40%) 2022-12-01T10:47:28.7140103Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102317.xml (deflated 41%) 2022-12-01T10:47:28.7140686Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102321.xml (deflated 40%) 2022-12-01T10:47:28.7141253Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102324.xml (deflated 40%) 2022-12-01T10:47:28.7141840Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102328.xml (deflated 40%) 2022-12-01T10:47:28.7142421Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102331.xml (deflated 40%) 2022-12-01T10:47:28.7143003Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102335.xml (deflated 40%) 2022-12-01T10:47:28.7143573Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102338.xml (deflated 40%) 2022-12-01T10:47:28.7144157Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102342.xml (deflated 40%) 2022-12-01T10:47:28.7144733Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102346.xml (deflated 40%) 2022-12-01T10:47:28.7145314Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102349.xml (deflated 40%) 2022-12-01T10:47:28.7145888Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102353.xml (deflated 40%) 2022-12-01T10:47:28.7146471Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102356.xml (deflated 40%) 2022-12-01T10:47:28.7147053Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102400.xml (deflated 41%) 2022-12-01T10:47:28.7147662Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102403.xml (deflated 41%) 2022-12-01T10:47:28.7148226Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102405.xml (deflated 40%) 2022-12-01T10:47:28.7148810Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102409.xml (deflated 41%) 2022-12-01T10:47:28.7149433Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102412.xml (deflated 40%) 2022-12-01T10:47:28.7150014Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102416.xml (deflated 40%) 2022-12-01T10:47:28.7150583Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102419.xml (deflated 40%) 2022-12-01T10:47:28.7151169Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102423.xml (deflated 40%) 2022-12-01T10:47:28.7151757Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102426.xml (deflated 40%) 2022-12-01T10:47:28.7152342Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102430.xml (deflated 40%) 2022-12-01T10:47:28.7152912Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102433.xml (deflated 40%) 2022-12-01T10:47:28.7153493Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102438.xml (deflated 40%) 2022-12-01T10:47:28.7154073Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102442.xml (deflated 41%) 2022-12-01T10:47:28.7154656Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102446.xml (deflated 40%) 2022-12-01T10:47:28.7155238Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102450.xml (deflated 41%) 2022-12-01T10:47:28.7155805Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102453.xml (deflated 40%) 2022-12-01T10:47:28.7156393Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102456.xml (deflated 40%) 2022-12-01T10:47:28.7156975Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102500.xml (deflated 42%) 2022-12-01T10:47:28.7157553Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102503.xml (deflated 40%) 2022-12-01T10:47:28.7158117Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102507.xml (deflated 40%) 2022-12-01T10:47:28.7158802Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102511.xml (deflated 40%) 2022-12-01T10:47:28.7159389Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102514.xml (deflated 40%) 2022-12-01T10:47:28.7159975Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102518.xml (deflated 40%) 2022-12-01T10:47:28.7160543Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102521.xml (deflated 40%) 2022-12-01T10:47:28.7161123Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102525.xml (deflated 40%) 2022-12-01T10:47:28.7161746Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102528.xml (deflated 40%) 2022-12-01T10:47:28.7162326Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102532.xml (deflated 40%) 2022-12-01T10:47:28.7162896Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102535.xml (deflated 40%) 2022-12-01T10:47:28.7163482Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102538.xml (deflated 40%) 2022-12-01T10:47:28.7164254Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102543.xml (deflated 40%) 2022-12-01T10:47:28.7164839Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102547.xml (deflated 44%) 2022-12-01T10:47:28.7165408Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102552.xml (deflated 40%) 2022-12-01T10:47:28.7166074Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102556.xml (deflated 40%) 2022-12-01T10:47:28.7166661Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102559.xml (deflated 40%) 2022-12-01T10:47:28.7167248Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102602.xml (deflated 40%) 2022-12-01T10:47:28.7167824Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102606.xml (deflated 41%) 2022-12-01T10:47:28.7168405Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102609.xml (deflated 40%) 2022-12-01T10:47:28.7168993Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102613.xml (deflated 40%) 2022-12-01T10:47:28.7169578Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102616.xml (deflated 40%) 2022-12-01T10:47:28.7170160Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102620.xml (deflated 41%) 2022-12-01T10:47:28.7170723Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102624.xml (deflated 40%) 2022-12-01T10:47:28.7171309Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102627.xml (deflated 41%) 2022-12-01T10:47:28.7171894Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102629.xml (deflated 40%) 2022-12-01T10:47:28.7172476Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102632.xml (deflated 41%) 2022-12-01T10:47:28.7173041Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102636.xml (deflated 40%) 2022-12-01T10:47:28.7173627Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102639.xml (deflated 40%) 2022-12-01T10:47:28.7174369Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102643.xml (deflated 40%) 2022-12-01T10:47:28.7175296Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102647.xml (deflated 40%) 2022-12-01T10:47:28.7176144Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102650.xml (deflated 40%) 2022-12-01T10:47:28.7176802Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102654.xml (deflated 41%) 2022-12-01T10:47:28.7177389Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102658.xml (deflated 41%) 2022-12-01T10:47:28.7177969Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102701.xml (deflated 40%) 2022-12-01T10:47:28.7178537Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102705.xml (deflated 40%) 2022-12-01T10:47:28.7179176Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102708.xml (deflated 40%) 2022-12-01T10:47:28.7179763Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102712.xml (deflated 40%) 2022-12-01T10:47:28.7180344Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102715.xml (deflated 40%) 2022-12-01T10:47:28.7180915Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102719.xml (deflated 40%) 2022-12-01T10:47:28.7181500Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102722.xml (deflated 40%) 2022-12-01T10:47:28.7182081Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102726.xml (deflated 40%) 2022-12-01T10:47:28.7182663Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102729.xml (deflated 40%) 2022-12-01T10:47:28.7183229Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102733.xml (deflated 40%) 2022-12-01T10:47:28.7183806Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102736.xml (deflated 40%) 2022-12-01T10:47:28.7184392Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102740.xml (deflated 41%) 2022-12-01T10:47:28.7184972Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102744.xml (deflated 40%) 2022-12-01T10:47:28.7185537Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102747.xml (deflated 40%) 2022-12-01T10:47:28.7186115Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102751.xml (deflated 41%) 2022-12-01T10:47:28.7186696Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102754.xml (deflated 39%) 2022-12-01T10:47:28.7187274Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102758.xml (deflated 40%) 2022-12-01T10:47:28.7187846Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102801.xml (deflated 40%) 2022-12-01T10:47:28.7188426Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102805.xml (deflated 40%) 2022-12-01T10:47:28.7189008Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102809.xml (deflated 41%) 2022-12-01T10:47:28.7189590Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102812.xml (deflated 41%) 2022-12-01T10:47:28.7190168Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102816.xml (deflated 40%) 2022-12-01T10:47:28.7190734Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102819.xml (deflated 41%) 2022-12-01T10:47:28.7191369Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102823.xml (deflated 41%) 2022-12-01T10:47:28.7191957Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102826.xml (deflated 41%) 2022-12-01T10:47:28.7192537Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102830.xml (deflated 41%) 2022-12-01T10:47:28.7193128Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102833.xml (deflated 40%) 2022-12-01T10:47:28.7193716Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102837.xml (deflated 40%) 2022-12-01T10:47:28.7194300Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102840.xml (deflated 40%) 2022-12-01T10:47:28.7194880Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102844.xml (deflated 40%) 2022-12-01T10:47:28.7195443Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102847.xml (deflated 40%) 2022-12-01T10:47:28.7196025Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102854.xml (deflated 40%) 2022-12-01T10:47:28.7196610Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102900.xml (deflated 41%) 2022-12-01T10:47:28.7197191Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102904.xml (deflated 40%) 2022-12-01T10:47:28.7197757Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102907.xml (deflated 40%) 2022-12-01T10:47:28.7198484Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102911.xml (deflated 40%) 2022-12-01T10:47:28.7199075Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102915.xml (deflated 40%) 2022-12-01T10:47:28.7199658Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102921.xml (deflated 40%) 2022-12-01T10:47:28.7200225Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102928.xml (deflated 40%) 2022-12-01T10:47:28.7200811Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102931.xml (deflated 40%) 2022-12-01T10:47:28.7201397Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102935.xml (deflated 40%) 2022-12-01T10:47:28.7201980Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102939.xml (deflated 40%) 2022-12-01T10:47:28.7202547Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102942.xml (deflated 40%) 2022-12-01T10:47:28.7203127Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102946.xml (deflated 40%) 2022-12-01T10:47:28.7203712Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102949.xml (deflated 40%) 2022-12-01T10:47:28.7204300Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102953.xml (deflated 40%) 2022-12-01T10:47:28.7204867Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201102956.xml (deflated 40%) 2022-12-01T10:47:28.7205507Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103002.xml (deflated 40%) 2022-12-01T10:47:28.7206150Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103008.xml (deflated 40%) 2022-12-01T10:47:28.7206735Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103011.xml (deflated 41%) 2022-12-01T10:47:28.7207304Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103015.xml (deflated 40%) 2022-12-01T10:47:28.7207933Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103018.xml (deflated 41%) 2022-12-01T10:47:28.7208521Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103022.xml (deflated 41%) 2022-12-01T10:47:28.7209102Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103025.xml (deflated 40%) 2022-12-01T10:47:28.7209669Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103029.xml (deflated 40%) 2022-12-01T10:47:28.7210244Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103032.xml (deflated 40%) 2022-12-01T10:47:28.7210829Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103036.xml (deflated 40%) 2022-12-01T10:47:28.7211411Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103040.xml (deflated 40%) 2022-12-01T10:47:28.7211990Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103043.xml (deflated 40%) 2022-12-01T10:47:28.7212558Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103047.xml (deflated 40%) 2022-12-01T10:47:28.7213141Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103050.xml (deflated 40%) 2022-12-01T10:47:28.7213722Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103054.xml (deflated 40%) 2022-12-01T10:47:28.7214300Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103058.xml (deflated 41%) 2022-12-01T10:47:28.7214866Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103101.xml (deflated 41%) 2022-12-01T10:47:28.7215451Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103105.xml (deflated 41%) 2022-12-01T10:47:28.7216032Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103108.xml (deflated 40%) 2022-12-01T10:47:28.7216617Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103112.xml (deflated 41%) 2022-12-01T10:47:28.7217181Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103115.xml (deflated 41%) 2022-12-01T10:47:28.7217767Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103119.xml (deflated 41%) 2022-12-01T10:47:28.7218351Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103122.xml (deflated 41%) 2022-12-01T10:47:28.7218932Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103126.xml (deflated 40%) 2022-12-01T10:47:28.7219495Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103129.xml (deflated 40%) 2022-12-01T10:47:28.7220107Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103133.xml (deflated 41%) 2022-12-01T10:47:28.7220694Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103137.xml (deflated 40%) 2022-12-01T10:47:28.7221276Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103140.xml (deflated 41%) 2022-12-01T10:47:28.7221862Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103144.xml (deflated 40%) 2022-12-01T10:47:28.7222447Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103147.xml (deflated 40%) 2022-12-01T10:47:28.7223029Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103151.xml (deflated 40%) 2022-12-01T10:47:28.7223608Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103155.xml (deflated 40%) 2022-12-01T10:47:28.7224176Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103158.xml (deflated 41%) 2022-12-01T10:47:28.7224752Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103202.xml (deflated 41%) 2022-12-01T10:47:28.7225342Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103206.xml (deflated 40%) 2022-12-01T10:47:28.7225920Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103211.xml (deflated 40%) 2022-12-01T10:47:28.7226490Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103215.xml (deflated 41%) 2022-12-01T10:47:28.7227072Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103218.xml (deflated 41%) 2022-12-01T10:47:28.7227655Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103228.xml (deflated 40%) 2022-12-01T10:47:28.7228237Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103232.xml (deflated 40%) 2022-12-01T10:47:28.7228803Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103235.xml (deflated 40%) 2022-12-01T10:47:28.7229375Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103239.xml (deflated 40%) 2022-12-01T10:47:28.7229961Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103242.xml (deflated 40%) 2022-12-01T10:47:28.7230543Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103246.xml (deflated 40%) 2022-12-01T10:47:28.7231122Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103249.xml (deflated 40%) 2022-12-01T10:47:28.7231688Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103253.xml (deflated 40%) 2022-12-01T10:47:28.7232274Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103257.xml (deflated 41%) 2022-12-01T10:47:28.7232857Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103300.xml (deflated 41%) 2022-12-01T10:47:28.7233433Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103304.xml (deflated 40%) 2022-12-01T10:47:28.7234027Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103307.xml (deflated 40%) 2022-12-01T10:47:28.7234611Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103311.xml (deflated 41%) 2022-12-01T10:47:28.7235190Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103316.xml (deflated 40%) 2022-12-01T10:47:28.7235801Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103320.xml (deflated 40%) 2022-12-01T10:47:28.7236367Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103324.xml (deflated 40%) 2022-12-01T10:47:28.7236951Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103327.xml (deflated 40%) 2022-12-01T10:47:28.7237535Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103331.xml (deflated 40%) 2022-12-01T10:47:28.7238113Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103335.xml (deflated 40%) 2022-12-01T10:47:28.7238806Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103339.xml (deflated 40%) 2022-12-01T10:47:28.7239389Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103343.xml (deflated 40%) 2022-12-01T10:47:28.7239974Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103346.xml (deflated 40%) 2022-12-01T10:47:28.7240555Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103350.xml (deflated 41%) 2022-12-01T10:47:28.7241123Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103353.xml (deflated 41%) 2022-12-01T10:47:28.7241709Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103357.xml (deflated 40%) 2022-12-01T10:47:28.7242286Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103400.xml (deflated 42%) 2022-12-01T10:47:28.7242867Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103404.xml (deflated 41%) 2022-12-01T10:47:28.7243429Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103408.xml (deflated 40%) 2022-12-01T10:47:28.7244007Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103411.xml (deflated 41%) 2022-12-01T10:47:28.7244592Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103415.xml (deflated 40%) 2022-12-01T10:47:28.7245173Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103418.xml (deflated 40%) 2022-12-01T10:47:28.7245783Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103422.xml (deflated 40%) 2022-12-01T10:47:28.7246368Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103425.xml (deflated 40%) 2022-12-01T10:47:28.7246952Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103429.xml (deflated 40%) 2022-12-01T10:47:28.7247532Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103433.xml (deflated 40%) 2022-12-01T10:47:28.7248153Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103436.xml (deflated 40%) 2022-12-01T10:47:28.7248730Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103440.xml (deflated 40%) 2022-12-01T10:47:28.7249311Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103443.xml (deflated 40%) 2022-12-01T10:47:28.7249894Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103447.xml (deflated 40%) 2022-12-01T10:47:28.7250490Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103450.xml (deflated 40%) 2022-12-01T10:47:28.7251071Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103454.xml (deflated 40%) 2022-12-01T10:47:28.7251655Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103458.xml (deflated 40%) 2022-12-01T10:47:28.7252235Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103501.xml (deflated 40%) 2022-12-01T10:47:28.7252811Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103505.xml (deflated 40%) 2022-12-01T10:47:28.7253375Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103509.xml (deflated 40%) 2022-12-01T10:47:28.7253959Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103512.xml (deflated 40%) 2022-12-01T10:47:28.7254538Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103516.xml (deflated 40%) 2022-12-01T10:47:28.7255117Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103519.xml (deflated 40%) 2022-12-01T10:47:28.7255688Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103524.xml (deflated 40%) 2022-12-01T10:47:28.7256273Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103527.xml (deflated 40%) 2022-12-01T10:47:28.7256851Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103531.xml (deflated 40%) 2022-12-01T10:47:28.7257432Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103535.xml (deflated 40%) 2022-12-01T10:47:28.7257993Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103540.xml (deflated 40%) 2022-12-01T10:47:28.7258578Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103543.xml (deflated 40%) 2022-12-01T10:47:28.7259162Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103547.xml (deflated 40%) 2022-12-01T10:47:28.7259740Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103550.xml (deflated 40%) 2022-12-01T10:47:28.7260305Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221201103554.xml (deflated 40%) 2022-12-01T10:47:28.7260984Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103557.xml (deflated 44%) 2022-12-01T10:47:28.7261736Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103601.xml (deflated 44%) 2022-12-01T10:47:28.7262516Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103604.xml (deflated 44%) 2022-12-01T10:47:28.7263255Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103608.xml (deflated 44%) 2022-12-01T10:47:28.7264001Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103612.xml (deflated 44%) 2022-12-01T10:47:28.7264782Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103615.xml (deflated 44%) 2022-12-01T10:47:28.7265535Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103619.xml (deflated 43%) 2022-12-01T10:47:28.7266286Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103623.xml (deflated 44%) 2022-12-01T10:47:28.7267021Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103626.xml (deflated 44%) 2022-12-01T10:47:28.7267771Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103630.xml (deflated 44%) 2022-12-01T10:47:28.7268517Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103634.xml (deflated 44%) 2022-12-01T10:47:28.7269267Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103637.xml (deflated 44%) 2022-12-01T10:47:28.7270001Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103641.xml (deflated 44%) 2022-12-01T10:47:28.7270750Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103645.xml (deflated 44%) 2022-12-01T10:47:28.7271496Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103648.xml (deflated 44%) 2022-12-01T10:47:28.7272245Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103652.xml (deflated 43%) 2022-12-01T10:47:28.7272988Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103656.xml (deflated 43%) 2022-12-01T10:47:28.7273723Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103700.xml (deflated 44%) 2022-12-01T10:47:28.7274471Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103703.xml (deflated 44%) 2022-12-01T10:47:28.7275221Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103707.xml (deflated 44%) 2022-12-01T10:47:28.7275966Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103711.xml (deflated 44%) 2022-12-01T10:47:28.7276695Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103715.xml (deflated 44%) 2022-12-01T10:47:28.7277439Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103719.xml (deflated 44%) 2022-12-01T10:47:28.7278286Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103723.xml (deflated 44%) 2022-12-01T10:47:28.7279098Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103726.xml (deflated 44%) 2022-12-01T10:47:28.7279829Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103731.xml (deflated 44%) 2022-12-01T10:47:28.7280581Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103735.xml (deflated 44%) 2022-12-01T10:47:28.7281358Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103739.xml (deflated 44%) 2022-12-01T10:47:28.7282106Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103743.xml (deflated 44%) 2022-12-01T10:47:28.7282854Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103746.xml (deflated 44%) 2022-12-01T10:47:28.7283585Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221201103750.xml (deflated 45%) 2022-12-01T10:47:28.7284300Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103754.xml (deflated 43%) 2022-12-01T10:47:28.7284992Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103758.xml (deflated 43%) 2022-12-01T10:47:28.7285672Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103801.xml (deflated 43%) 2022-12-01T10:47:28.7286402Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103805.xml (deflated 43%) 2022-12-01T10:47:28.7287090Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103807.xml (deflated 43%) 2022-12-01T10:47:28.7287769Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103810.xml (deflated 44%) 2022-12-01T10:47:28.7288454Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103814.xml (deflated 43%) 2022-12-01T10:47:28.7289136Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103817.xml (deflated 43%) 2022-12-01T10:47:28.7289803Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103821.xml (deflated 43%) 2022-12-01T10:47:28.7290484Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103825.xml (deflated 43%) 2022-12-01T10:47:28.7291157Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103828.xml (deflated 43%) 2022-12-01T10:47:28.7291840Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103832.xml (deflated 43%) 2022-12-01T10:47:28.7292503Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103836.xml (deflated 43%) 2022-12-01T10:47:28.7293177Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103840.xml (deflated 43%) 2022-12-01T10:47:28.7293857Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103843.xml (deflated 43%) 2022-12-01T10:47:28.7294570Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103847.xml (deflated 43%) 2022-12-01T10:47:28.7295235Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103851.xml (deflated 43%) 2022-12-01T10:47:28.7295908Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103855.xml (deflated 43%) 2022-12-01T10:47:28.7296614Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103858.xml (deflated 43%) 2022-12-01T10:47:28.7297296Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103902.xml (deflated 43%) 2022-12-01T10:47:28.7297959Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103906.xml (deflated 43%) 2022-12-01T10:47:28.7298638Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103909.xml (deflated 43%) 2022-12-01T10:47:28.7299316Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103913.xml (deflated 44%) 2022-12-01T10:47:28.7299989Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103917.xml (deflated 43%) 2022-12-01T10:47:28.7300668Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103922.xml (deflated 43%) 2022-12-01T10:47:28.7301338Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103928.xml (deflated 43%) 2022-12-01T10:47:28.7302024Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103945.xml (deflated 43%) 2022-12-01T10:47:28.7302705Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103948.xml (deflated 44%) 2022-12-01T10:47:28.7303385Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103952.xml (deflated 43%) 2022-12-01T10:47:28.7304054Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201103956.xml (deflated 44%) 2022-12-01T10:47:28.7304736Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104000.xml (deflated 44%) 2022-12-01T10:47:28.7305412Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104003.xml (deflated 43%) 2022-12-01T10:47:28.7306094Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104007.xml (deflated 43%) 2022-12-01T10:47:28.7306761Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104011.xml (deflated 43%) 2022-12-01T10:47:28.7307444Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104014.xml (deflated 44%) 2022-12-01T10:47:28.7308123Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104018.xml (deflated 44%) 2022-12-01T10:47:28.7308811Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104021.xml (deflated 43%) 2022-12-01T10:47:28.7309517Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104025.xml (deflated 43%) 2022-12-01T10:47:28.7310180Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221201104029.xml (deflated 43%) 2022-12-01T10:47:28.7310888Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221201104032.xml (deflated 44%) 2022-12-01T10:47:28.7311644Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221201104036.xml (deflated 43%) 2022-12-01T10:47:28.7312371Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221201104040.xml (deflated 43%) 2022-12-01T10:47:28.7313061Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20221201104045.xml (deflated 42%) 2022-12-01T10:47:28.7313742Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20221201104055.xml (deflated 42%) 2022-12-01T10:47:28.7314391Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104105.xml (deflated 41%) 2022-12-01T10:47:28.7315008Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104114.xml (deflated 41%) 2022-12-01T10:47:28.7315610Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104123.xml (deflated 41%) 2022-12-01T10:47:28.7316226Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104137.xml (deflated 42%) 2022-12-01T10:47:28.7316836Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104147.xml (deflated 41%) 2022-12-01T10:47:28.7317454Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104150.xml (deflated 42%) 2022-12-01T10:47:28.7318044Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104159.xml (deflated 41%) 2022-12-01T10:47:28.7318811Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104208.xml (deflated 41%) 2022-12-01T10:47:28.7319426Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104213.xml (deflated 41%) 2022-12-01T10:47:28.7320034Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104223.xml (deflated 41%) 2022-12-01T10:47:28.7320640Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104232.xml (deflated 41%) 2022-12-01T10:47:28.7321232Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104241.xml (deflated 41%) 2022-12-01T10:47:28.7321840Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104246.xml (deflated 42%) 2022-12-01T10:47:28.7322444Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104303.xml (deflated 41%) 2022-12-01T10:47:28.7323052Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104314.xml (deflated 41%) 2022-12-01T10:47:28.7323641Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104319.xml (deflated 41%) 2022-12-01T10:47:28.7324249Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104329.xml (deflated 41%) 2022-12-01T10:47:28.7324916Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104335.xml (deflated 41%) 2022-12-01T10:47:28.7325528Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104345.xml (deflated 41%) 2022-12-01T10:47:28.7326178Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221201104354.xml (deflated 41%) 2022-12-01T10:47:28.7326845Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104403.xml (deflated 41%) 2022-12-01T10:47:28.7327493Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104413.xml (deflated 41%) 2022-12-01T10:47:28.7328126Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104422.xml (deflated 42%) 2022-12-01T10:47:28.7328752Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104432.xml (deflated 42%) 2022-12-01T10:47:28.7329381Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104437.xml (deflated 42%) 2022-12-01T10:47:28.7330017Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221201104445.xml (deflated 42%) 2022-12-01T10:47:28.7330635Z adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20221201104453.xml (deflated 86%) 2022-12-01T10:47:28.7331218Z adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20221201104453.xml (deflated 86%) 2022-12-01T10:47:28.7331885Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221201104551.xml (deflated 44%) 2022-12-01T10:47:28.7332614Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221201104552.xml (deflated 44%) 2022-12-01T10:47:28.7333330Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221201104554.xml (deflated 44%) 2022-12-01T10:47:28.7333993Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221201104555.xml (deflated 41%) 2022-12-01T10:47:28.7334644Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221201104557.xml (deflated 42%) 2022-12-01T10:47:28.7335274Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221201104558.xml (deflated 42%) 2022-12-01T10:47:28.7335907Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221201104559.xml (deflated 42%) 2022-12-01T10:47:28.7336549Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104601.xml (deflated 43%) 2022-12-01T10:47:28.7337189Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104604.xml (deflated 44%) 2022-12-01T10:47:28.7337837Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104607.xml (deflated 43%) 2022-12-01T10:47:28.7338478Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104609.xml (deflated 43%) 2022-12-01T10:47:28.7339121Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104612.xml (deflated 44%) 2022-12-01T10:47:28.7339782Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104615.xml (deflated 43%) 2022-12-01T10:47:28.7340428Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104618.xml (deflated 43%) 2022-12-01T10:47:28.7341075Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221201104621.xml (deflated 43%) 2022-12-01T10:47:28.7341696Z adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221201104624.xml (deflated 80%) 2022-12-01T10:47:28.7342272Z adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221201104636.xml (deflated 69%) 2022-12-01T10:47:28.7342896Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20221201104643.xml (deflated 66%) 2022-12-01T10:47:28.7343516Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20221201104643.xml (deflated 69%) 2022-12-01T10:47:28.7344201Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20221201104643.xml (deflated 66%) 2022-12-01T10:47:28.7344827Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221201104647.xml (deflated 63%) 2022-12-01T10:47:28.7345393Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221201104647.xml (deflated 69%) 2022-12-01T10:47:28.7345972Z adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221201104649.xml (deflated 63%) 2022-12-01T10:47:28.7346558Z adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20221201104650.xml (deflated 43%) 2022-12-01T10:47:28.7347156Z adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221201104652.xml (deflated 78%) 2022-12-01T10:47:28.7347766Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221201104653.xml (deflated 78%) 2022-12-01T10:47:28.7348372Z adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20221201104654.xml (deflated 69%) 2022-12-01T10:47:28.7348994Z adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20221201104654.xml (deflated 62%) 2022-12-01T10:47:28.7349633Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221201104655.xml (deflated 71%) 2022-12-01T10:47:28.7350267Z adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20221201104656.xml (deflated 61%) 2022-12-01T10:47:28.7350906Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221201104657.xml (deflated 62%) 2022-12-01T10:47:28.7351523Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221201104658.xml (deflated 63%) 2022-12-01T10:47:28.7352161Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20221201104700.xml (deflated 62%) 2022-12-01T10:47:28.7352797Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20221201104701.xml (deflated 61%) 2022-12-01T10:47:28.7353270Z adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%) 2022-12-01T10:47:28.7371231Z ##[group]Run # Remove any previous test reports if they exist 2022-12-01T10:47:28.7371590Z # Remove any previous test reports if they exist 2022-12-01T10:47:28.7371824Z rm -f usage-log-*.zip 2022-12-01T10:47:28.7372097Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2022-12-01T10:47:28.7372384Z # so check to see if the file exists first 2022-12-01T10:47:28.7372603Z if [ -f 'usage_log.txt' ]; then 2022-12-01T10:47:28.7372846Z  zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt' 2022-12-01T10:47:28.7373056Z fi 2022-12-01T10:47:28.7383782Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:47:28.7383995Z env: 2022-12-01T10:47:28.7384231Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:28.7384475Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_9818325812 2022-12-01T10:47:28.7384714Z ##[endgroup] 2022-12-01T10:47:28.7728636Z adding: usage_log.txt (deflated 94%) 2022-12-01T10:47:28.7766151Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-12-01T10:47:28.7766392Z with: 2022-12-01T10:47:28.7766650Z s3-prefix: pytorch/pytorch/3591403534/1/artifact 2022-12-01T10:47:28.7766997Z retention-days: 14 2022-12-01T10:47:28.7767250Z if-no-files-found: warn 2022-12-01T10:47:28.7767465Z path: test-jsons-*.zip 2022-12-01T10:47:28.7767704Z name: artifact 2022-12-01T10:47:28.7767953Z s3-bucket: gha-artifacts 2022-12-01T10:47:28.7768180Z region: us-east-1 2022-12-01T10:47:28.7768399Z env: 2022-12-01T10:47:28.7768624Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:28.7768830Z ##[endgroup] 2022-12-01T10:47:29.1100656Z NOTE: s3-prefix specified, ignoring name parameter 2022-12-01T10:47:29.1101171Z With the provided path, there will be 1 file uploaded 2022-12-01T10:47:29.1101680Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact 2022-12-01T10:47:29.1108282Z Starting upload of test-jsons-test-distributed-1-2-linux.2xlarge_9818325812.zip 2022-12-01T10:47:29.2361101Z Finished upload of test-jsons-test-distributed-1-2-linux.2xlarge_9818325812.zip 2022-12-01T10:47:29.2483958Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-12-01T10:47:29.2484157Z with: 2022-12-01T10:47:29.2484360Z s3-prefix: pytorch/pytorch/3591403534/1/artifact 2022-12-01T10:47:29.2484578Z retention-days: 14 2022-12-01T10:47:29.2484759Z if-no-files-found: error 2022-12-01T10:47:29.2484959Z path: test-reports-*.zip 2022-12-01T10:47:29.2485147Z name: artifact 2022-12-01T10:47:29.2485314Z s3-bucket: gha-artifacts 2022-12-01T10:47:29.2485500Z region: us-east-1 2022-12-01T10:47:29.2485663Z env: 2022-12-01T10:47:29.2485893Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:29.2486078Z ##[endgroup] 2022-12-01T10:47:29.5828432Z NOTE: s3-prefix specified, ignoring name parameter 2022-12-01T10:47:29.5828926Z With the provided path, there will be 1 file uploaded 2022-12-01T10:47:29.5829419Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact 2022-12-01T10:47:29.5836024Z Starting upload of test-reports-test-distributed-1-2-linux.2xlarge_9818325812.zip 2022-12-01T10:47:29.7350418Z Finished upload of test-reports-test-distributed-1-2-linux.2xlarge_9818325812.zip 2022-12-01T10:47:29.7467842Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-12-01T10:47:29.7468062Z with: 2022-12-01T10:47:29.7468261Z s3-prefix: pytorch/pytorch/3591403534/1/artifact 2022-12-01T10:47:29.7468482Z retention-days: 14 2022-12-01T10:47:29.7468663Z if-no-files-found: ignore 2022-12-01T10:47:29.7468863Z path: usage-log-*.zip 2022-12-01T10:47:29.7469041Z name: artifact 2022-12-01T10:47:29.7469210Z s3-bucket: gha-artifacts 2022-12-01T10:47:29.7469402Z region: us-east-1 2022-12-01T10:47:29.7469566Z env: 2022-12-01T10:47:29.7469724Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:29.7469917Z ##[endgroup] 2022-12-01T10:47:30.0824571Z NOTE: s3-prefix specified, ignoring name parameter 2022-12-01T10:47:30.0825078Z With the provided path, there will be 1 file uploaded 2022-12-01T10:47:30.0825536Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact 2022-12-01T10:47:30.0833931Z Starting upload of usage-log-test-distributed-1-2-linux.2xlarge_9818325812.zip 2022-12-01T10:47:30.2345412Z Finished upload of usage-log-test-distributed-1-2-linux.2xlarge_9818325812.zip 2022-12-01T10:47:30.2470545Z ##[group]Run set -x 2022-12-01T10:47:30.2470746Z set -x 2022-12-01T10:47:30.2470969Z python3 -m pip install -r requirements.txt 2022-12-01T10:47:30.2471219Z python3 -m pip install boto3==1.19.12 2022-12-01T10:47:30.2471498Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-12-01T10:47:30.2482981Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:47:30.2483204Z env: 2022-12-01T10:47:30.2483368Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:30.2483573Z AWS_DEFAULT_REGION: us-east-1 2022-12-01T10:47:30.2483767Z BRANCH: pull/89997 2022-12-01T10:47:30.2483940Z TEST_CONFIG: distributed 2022-12-01T10:47:30.2484123Z SHARD_NUMBER: 1 2022-12-01T10:47:30.2484339Z BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7 2022-12-01T10:47:30.2484550Z PR_NUMBER: 89997 2022-12-01T10:47:30.2484739Z PYTORCH_RETRY_TEST_CASES: 1 2022-12-01T10:47:30.2484949Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-12-01T10:47:30.2485203Z SHA1: c13d400bffe90e16b96520bbc8a41a6f0c9cd584 2022-12-01T10:47:30.2485406Z TAG: 2022-12-01T10:47:30.2485575Z WORKFLOW_ID: 3591403534 2022-12-01T10:47:30.2485978Z GITHUB_TOKEN: *** 2022-12-01T10:47:30.2486156Z GHA_WORKFLOW_JOB_ID: 9818325812 2022-12-01T10:47:30.2486341Z ##[endgroup] 2022-12-01T10:47:30.2510958Z + python3 -m pip install -r requirements.txt 2022-12-01T10:47:30.4598474Z Defaulting to user installation because normal site-packages is not writeable 2022-12-01T10:47:30.4857005Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3) 2022-12-01T10:47:30.4884047Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4) 2022-12-01T10:47:30.4892367Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2) 2022-12-01T10:47:30.4900544Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.58.2) 2022-12-01T10:47:30.5262078Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6) 2022-12-01T10:47:30.5270339Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1) 2022-12-01T10:47:30.5347058Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0) 2022-12-01T10:47:30.5354682Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0) 2022-12-01T10:47:30.5571438Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3) 2022-12-01T10:47:30.5736517Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0) 2022-12-01T10:47:30.5744989Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6) 2022-12-01T10:47:30.5750665Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0) 2022-12-01T10:47:30.5760334Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1) 2022-12-01T10:47:30.5780979Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4) 2022-12-01T10:47:30.5796775Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0) 2022-12-01T10:47:30.5807251Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.4) 2022-12-01T10:47:30.5827090Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0) 2022-12-01T10:47:30.6070158Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24) 2022-12-01T10:47:30.6079194Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12) 2022-12-01T10:47:30.6097899Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4) 2022-12-01T10:47:30.6109192Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.13) 2022-12-01T10:47:30.6261710Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1) 2022-12-01T10:47:30.8120719Z + python3 -m pip install boto3==1.19.12 2022-12-01T10:47:31.0205101Z Defaulting to user installation because normal site-packages is not writeable 2022-12-01T10:47:31.0389842Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12) 2022-12-01T10:47:31.0438989Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2) 2022-12-01T10:47:31.0465681Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12) 2022-12-01T10:47:31.0515797Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0) 2022-12-01T10:47:31.0528751Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.13) 2022-12-01T10:47:31.0681829Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2) 2022-12-01T10:47:31.0700075Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2022-12-01T10:47:31.3555487Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-12-01T10:47:39.6131643Z [scribe] Scribe access token not provided, sending report via boto3... 2022-12-01T10:47:39.6132007Z 2022-12-01T10:47:39.6132435Z ----- Historic stats comparison result ------ 2022-12-01T10:47:39.6132745Z 2022-12-01T10:47:39.6133003Z job: linux-focal-py3.7-gcc7 2022-12-01T10:47:39.6133427Z commit: c13d400bffe90e16b96520bbc8a41a6f0c9cd584 2022-12-01T10:47:39.6133693Z 2022-12-01T10:47:39.6133961Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-12-01T10:47:39.6134280Z 2022-12-01T10:47:39.6134410Z : (master) 2022-12-01T10:47:39.6134705Z | 2022-12-01T10:47:39.6135082Z | * c13d400bff (HEAD) total time 1598.75s 2022-12-01T10:47:39.6135445Z | | 2022-12-01T10:47:39.6135747Z | : (59 commits) 2022-12-01T10:47:39.6136048Z |/ 2022-12-01T10:47:39.6136602Z * 67eb2d5952 (base) 12 reports, total time 2500.30s ± 1654.93s 2022-12-01T10:47:39.6137186Z * 1c5ca724f4 6 reports, total time 2466.99s ± 1724.34s 2022-12-01T10:47:39.6138029Z * 9d6109c4b0 6 reports, total time 2484.54s ± 1728.75s 2022-12-01T10:47:39.6138705Z * 736adc0808 6 reports, total time 2481.20s ± 1744.38s 2022-12-01T10:47:39.6139270Z * a348975e00 6 reports, total time 2497.04s ± 1746.31s 2022-12-01T10:47:39.6139803Z * db13049b88 6 reports, total time 2514.30s ± 1779.23s 2022-12-01T10:47:39.6140354Z * d07b85393a 6 reports, total time 2478.62s ± 1754.66s 2022-12-01T10:47:39.6140908Z * ac25c210e5 6 reports, total time 2568.96s ± 1821.46s 2022-12-01T10:47:39.6141447Z * 2355b6256b 6 reports, total time 2536.34s ± 1800.59s 2022-12-01T10:47:39.6142153Z * 4f95f7ae9b 6 reports, total time 2504.50s ± 1759.28s 2022-12-01T10:47:39.6142530Z | 2022-12-01T10:47:39.6142818Z : 2022-12-01T10:47:39.6142981Z 2022-12-01T10:47:39.6143212Z Removed (across 816 suites) 0 tests, totaling 0.00s 2022-12-01T10:47:39.6143698Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-12-01T10:47:39.6144202Z Added (across 92 suites) 998 tests, totaling +1598.75s 2022-12-01T10:47:39.6654483Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2022-12-01T10:47:39.6654730Z with: 2022-12-01T10:47:39.6654881Z env: 2022-12-01T10:47:39.6655040Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:39.6655222Z ##[endgroup] 2022-12-01T10:47:39.6669873Z ##[group]Run set -eou pipefail 2022-12-01T10:47:39.6670098Z set -eou pipefail 2022-12-01T10:47:39.6670280Z  2022-12-01T10:47:39.6670502Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2022-12-01T10:47:39.6670755Z for _ in $(seq 1440); do 2022-12-01T10:47:39.6670975Z  # Break if no ssh session exists anymore 2022-12-01T10:47:39.6671180Z  if [ "$(who)" = "" ]; then 2022-12-01T10:47:39.6671364Z  break 2022-12-01T10:47:39.6671532Z  fi 2022-12-01T10:47:39.6671700Z  echo "." 2022-12-01T10:47:39.6671867Z  sleep 5 2022-12-01T10:47:39.6672033Z done 2022-12-01T10:47:39.6683227Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:47:39.6683457Z env: 2022-12-01T10:47:39.6683617Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:39.6683806Z ##[endgroup] 2022-12-01T10:47:39.6707887Z Holding runner for 2 hours until all ssh sessions have logged out 2022-12-01T10:47:39.6745537Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2022-12-01T10:47:39.6745855Z # ignore expansion of "docker ps -q" since it could be empty 2022-12-01T10:47:39.6746107Z # shellcheck disable=SC2046 2022-12-01T10:47:39.6746323Z docker stop $(docker ps -q) || true 2022-12-01T10:47:39.6746553Z # Prune all of the docker images 2022-12-01T10:47:39.6746768Z docker system prune -af 2022-12-01T10:47:39.6757063Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-12-01T10:47:39.6757291Z env: 2022-12-01T10:47:39.6757470Z GIT_DEFAULT_BRANCH: master 2022-12-01T10:47:39.6757644Z ##[endgroup] 2022-12-01T10:47:40.0523281Z ad94a0d9dab5 2022-12-01T10:47:40.6024629Z Deleted Containers: 2022-12-01T10:47:40.6025132Z ad94a0d9dab526da42383dc844e6795f91cab8542f2f3aa7f61579dc8525ac0c 2022-12-01T10:47:40.6025337Z 2022-12-01T10:47:43.1491444Z Deleted Images: 2022-12-01T10:47:43.1492477Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b 2022-12-01T10:47:43.1493134Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:7d4daa1530ad815417dd92428a12643a4a8b0e93e820e2eb2d4e8818a4e57ba2 2022-12-01T10:47:43.1493616Z deleted: sha256:e8be7981a8238969daaef272b84b69737c5858187c9b10870da76dec899f6864 2022-12-01T10:47:43.1493944Z deleted: sha256:dd542825775450ac98a8f341e744c9b998cf9f91f0b678b696bdfdcc8197900b 2022-12-01T10:47:43.1494323Z deleted: sha256:c2f995bd392fa39048aba9ecdbe2ce96fb0da354f391973cab978f1fdc1bafe9 2022-12-01T10:47:43.1494871Z deleted: sha256:0ba22d6d32aea10bff37a734be2eabf0ca45365a8c84009c11c72697bbeb57ac 2022-12-01T10:47:43.1495255Z deleted: sha256:1641ef8553a9d06a751fe1c9fa58edae658b58be5e4452cf0e6c22fe7e9e9f7a 2022-12-01T10:47:43.1495581Z deleted: sha256:6fe08b57b8677fadf24df12c30c6b075caa4e95cff1e8121398e7712e7cc816b 2022-12-01T10:47:43.1495917Z deleted: sha256:b535a07f21c50706835dc1c6a9f6134e10f492c0a72ef3edd3d61eb8b057318a 2022-12-01T10:47:43.1496245Z deleted: sha256:ad51a4c1d9f796ae822ae9448a0439daec4dc5352ab73405d5170d8320d6569b 2022-12-01T10:47:43.1496583Z deleted: sha256:cc3572e58e1eadfed379b3421fc485a2fec838dbe2f03f9482a844387875539b 2022-12-01T10:47:43.1496889Z deleted: sha256:f31159505662112f25c640d83a5486c60dcea9996cfac1468ae7b74838077420 2022-12-01T10:47:43.1497200Z deleted: sha256:5d54ecd58fecab272c8614e2c443272f47c9245eaf4b5be502820db913ddd776 2022-12-01T10:47:43.1497545Z deleted: sha256:e8fcb42d7c24a5f6cd663f59d8fd82a2ad892e66046dc0d36ee4bdadf1282b52 2022-12-01T10:47:43.1497875Z deleted: sha256:934da62f4ce129178b2355504cdb61701bffc48d1bcf3b1ce0ca6e4c89bdfa31 2022-12-01T10:47:43.1498275Z deleted: sha256:8dccdf71b42bc4d3717c42c3d96812509d1a909bebd7848d0ddae3cc40170990 2022-12-01T10:47:43.1498619Z deleted: sha256:d7cb3c132d871b52eaf2533cfbc7ebca30405eff1b313986ce565b057fca56d8 2022-12-01T10:47:43.1498961Z deleted: sha256:a5fa381fecd94b52ee79c5c4ce30770402b16f7ee3dc77f7f1dccb386626b1ae 2022-12-01T10:47:43.1499291Z deleted: sha256:de8b14689d2a8b030e41a969f6faf30e9dee1048aac1c5d9a38a911b6af1ccca 2022-12-01T10:47:43.1499638Z deleted: sha256:ddcc5d19eb58848e898aedcc3a221bc40ebba2752ba3b7bd50babd76c1e225ae 2022-12-01T10:47:43.1499975Z deleted: sha256:a67b9c1632db819366358a5052f33ed59ff1e1cd03e0cc15e82d8c89a9417b97 2022-12-01T10:47:43.1500299Z deleted: sha256:a5fc4c6c0aaad2d5fcd0d4cc4cf194a8892115d9e7faf6c578e4826937a72d34 2022-12-01T10:47:43.1500618Z deleted: sha256:f33cf9181261127561bc276f483d4f9927a63395c4343028c589ac8905dd0cd7 2022-12-01T10:47:43.1500925Z deleted: sha256:4364081b1a5daf042c41fa0b07d65e0488c3a988dcea07cec40851d8bb481c01 2022-12-01T10:47:43.1501261Z deleted: sha256:2c4beb5d46c3bdc6af376c4a784f3ea2032feda5143dc9b56be143568b8d71cd 2022-12-01T10:47:43.1501561Z deleted: sha256:3e002fe51ea8377736f778797676c87d37546763f35bc5f14a32121200c634ae 2022-12-01T10:47:43.1501864Z deleted: sha256:d1e577d22ad4a397fc4186062a6a2fbb898c4fb7c325794f0760c084ac4e6831 2022-12-01T10:47:43.1502164Z deleted: sha256:0781305ea794027575e6910f20b6c90cc225056d0112fa7f36ac8008431f5791 2022-12-01T10:47:43.1502443Z deleted: sha256:79f50ef439b6855c9ca6d4be55467b692799f788388447976d8aab0a26d805f4 2022-12-01T10:47:43.1502756Z deleted: sha256:fcfdd0674d63c370d31701118409d5e8b4d9b5eaa36b410f032f5fccf29e4f11 2022-12-01T10:47:43.1503080Z deleted: sha256:c868e869e0efb64e8a908cf5a276657da1190183a5aa433a72e1fda6313e8841 2022-12-01T10:47:43.1503404Z deleted: sha256:0ac36be062e7b703a8ffa26b178cc9dd7c19c377bd781f6a369d50516bdca8cb 2022-12-01T10:47:43.1503715Z deleted: sha256:6c9628b9c75a72926209bea0ff44fb2efe306ecb21d57908bb514b1d81b5de8e 2022-12-01T10:47:43.1504044Z deleted: sha256:07c5d0507acce428b3637d5f713aed3fc6dd2dfad749e1961ec04f85d2a3e395 2022-12-01T10:47:43.1504379Z deleted: sha256:278ae5d4c6743b99236e1f2e5df4c346df5f068fa1bce0ba227861042971d612 2022-12-01T10:47:43.1504703Z deleted: sha256:d53c6cc3bfa4330c26ff3a73093bf35a7f28e7fc824782cfc1e00fcbc0390574 2022-12-01T10:47:43.1505040Z deleted: sha256:0a926d106d3065ecc114c8690aee4bab27d4f64ee81c96f90c4daf7a3c03fe7f 2022-12-01T10:47:43.1505367Z deleted: sha256:ab29088e02867cf532a2e8cfa0e1b0a1d13b428459232c7b1a4bfe961fd5a511 2022-12-01T10:47:43.1505680Z deleted: sha256:985cef3ba9e1ea2ed2249284a7478858f8463282738122494ab18c65fbbb7c7d 2022-12-01T10:47:43.1505985Z deleted: sha256:67ca28cabbfd7a80760061b52dbcfe8f5401b867526206d76c710a11cf9ab38e 2022-12-01T10:47:43.1506302Z deleted: sha256:2c9d0b64a3450004672e73e7ace7bc49da29654fd1d603ecf96e9161708726a3 2022-12-01T10:47:43.1506631Z deleted: sha256:efb1cb2fb03e227ccaa27426fae8f80bf1c082d14632685fff079b0c20111d3b 2022-12-01T10:47:43.1506984Z deleted: sha256:2bc3f10f3904b8fff87213a8a2325ab12e4048c0a19247b3d82886066b37a60d 2022-12-01T10:47:43.1507305Z deleted: sha256:adf0c04002f67cf46889421aed44a0e23618d009f3bbfb6aa94e3fced3cdb131 2022-12-01T10:47:43.1507619Z deleted: sha256:5b43511c57d66e58ff42011707835b8cc6971ed316454cfd950e0a5a12138631 2022-12-01T10:47:43.1507929Z deleted: sha256:ad9d96076f7f0bce4820b7828780dbf5328c1c45b9b6c93cfc21fa6c78e66765 2022-12-01T10:47:43.1508232Z deleted: sha256:545999a778d60ca8c6f4e76973a1c707cc626147f1e4d83eebbad6a57101cce0 2022-12-01T10:47:43.1508542Z deleted: sha256:cfcad11f69b44005150106d3108821c82a41765afbeb809208c1e0f49d5b6135 2022-12-01T10:47:43.1508879Z deleted: sha256:f7c8d5bcafe4ebe33f704b703a618803e7ade038c4cef7b4c3b520e9a100cd53 2022-12-01T10:47:43.1509211Z deleted: sha256:4bad106558a8cdd7564af2e1486b545ff1018677d73a8dcb029ecea2ad5afa9a 2022-12-01T10:47:43.1509603Z deleted: sha256:b01943462df81aa35c0f222112f8a7a13b09ec19be40907c2a81fded5e894e14 2022-12-01T10:47:43.1509955Z deleted: sha256:74a161fbdc63f2caa195797d69cca089d2f2ec781d7e7dc78fac88f57c610741 2022-12-01T10:47:43.1510399Z deleted: sha256:d94fdd2bb4227aed66703d740136d3bba2d45ac1c65930624fc2e7ab50021822 2022-12-01T10:47:43.1510717Z deleted: sha256:b348cd6909427063b9b36464ac9e6edb7d42ade2f164c5a98ee3ec37b1074e72 2022-12-01T10:47:43.1511036Z deleted: sha256:0fba67987a5ab74fca03bb2b1d623325a4c40e0e42a74379322b082612b8b445 2022-12-01T10:47:43.1511348Z deleted: sha256:030e74143974068ec1ab10e03d50f82455d36264aa70c98bd0fad278e4425d28 2022-12-01T10:47:43.1511749Z deleted: sha256:b7fcb4ae2181c6c159bcc2f4c7b624150a8245dbdeee9d8601d008b5cd8be8fd 2022-12-01T10:47:43.1512072Z deleted: sha256:4472f94e146cc542a9111182b122e9a9c93a8f7022a48ce3c94b600be8fcdfaa 2022-12-01T10:47:43.1512392Z deleted: sha256:33bc8c1ba43ca724221fd1a8c992a420b1d8de4c85372defdc409e5c88c005b9 2022-12-01T10:47:43.1512718Z deleted: sha256:7070a4ad8ac000d6910f8bab3e00671f9ea4e7094064da08eb1892332166b92d 2022-12-01T10:47:43.1513017Z deleted: sha256:b40ed86654e59e1012e1716d5384910f8c3bb58274b7b00fca564a53e9897ba3 2022-12-01T10:47:43.1513190Z 2022-12-01T10:47:43.1513477Z Total reclaimed space: 6.712GB 2022-12-01T10:47:43.1564163Z Post job cleanup. 2022-12-01T10:47:43.1593196Z Post job cleanup. 2022-12-01T10:47:43.2579374Z [command]/usr/bin/git version 2022-12-01T10:47:43.2621312Z git version 2.37.1 2022-12-01T10:47:43.2661153Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/caf1c47b-20e2-4e37-b5e4-3e3c662a1347' before making global git config changes 2022-12-01T10:47:43.2661880Z Adding repository directory to the temporary git global config as a safe directory 2022-12-01T10:47:43.2666480Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-12-01T10:47:43.2702446Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-12-01T10:47:43.2732030Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-12-01T10:47:43.2986356Z Entering 'android/libs/fbjni' 2022-12-01T10:47:43.3021836Z Entering 'third_party/FP16' 2022-12-01T10:47:43.3059746Z Entering 'third_party/FXdiv' 2022-12-01T10:47:43.3094946Z Entering 'third_party/NNPACK' 2022-12-01T10:47:43.3130088Z Entering 'third_party/QNNPACK' 2022-12-01T10:47:43.3165730Z Entering 'third_party/VulkanMemoryAllocator' 2022-12-01T10:47:43.3199801Z Entering 'third_party/XNNPACK' 2022-12-01T10:47:43.3244975Z Entering 'third_party/benchmark' 2022-12-01T10:47:43.3278807Z Entering 'third_party/cpuinfo' 2022-12-01T10:47:43.3313782Z Entering 'third_party/cub' 2022-12-01T10:47:43.3349133Z Entering 'third_party/cudnn_frontend' 2022-12-01T10:47:43.3388141Z Entering 'third_party/cutlass' 2022-12-01T10:47:43.3430226Z Entering 'third_party/eigen' 2022-12-01T10:47:43.3467705Z Entering 'third_party/fbgemm' 2022-12-01T10:47:43.3501741Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-12-01T10:47:43.3534805Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-12-01T10:47:43.3571367Z Entering 'third_party/fbgemm/third_party/googletest' 2022-12-01T10:47:43.3604603Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-12-01T10:47:43.3639034Z Entering 'third_party/flatbuffers' 2022-12-01T10:47:43.3675342Z Entering 'third_party/fmt' 2022-12-01T10:47:43.3708704Z Entering 'third_party/foxi' 2022-12-01T10:47:43.3743407Z Entering 'third_party/gemmlowp/gemmlowp' 2022-12-01T10:47:43.3777190Z Entering 'third_party/gloo' 2022-12-01T10:47:43.3810504Z Entering 'third_party/googletest' 2022-12-01T10:47:43.3843865Z Entering 'third_party/ideep' 2022-12-01T10:47:43.3877226Z Entering 'third_party/ideep/mkl-dnn' 2022-12-01T10:47:43.3911134Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-12-01T10:47:43.3951781Z Entering 'third_party/ios-cmake' 2022-12-01T10:47:43.3985950Z Entering 'third_party/ittapi' 2022-12-01T10:47:43.4019961Z Entering 'third_party/kineto' 2022-12-01T10:47:43.4054682Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-12-01T10:47:43.4088512Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-12-01T10:47:43.4122825Z Entering 'third_party/nccl/nccl' 2022-12-01T10:47:43.4156289Z Entering 'third_party/neon2sse' 2022-12-01T10:47:43.4190406Z Entering 'third_party/nlohmann' 2022-12-01T10:47:43.4224805Z Entering 'third_party/onnx' 2022-12-01T10:47:43.4270517Z Entering 'third_party/onnx/third_party/benchmark' 2022-12-01T10:47:43.4303911Z Entering 'third_party/onnx/third_party/pybind11' 2022-12-01T10:47:43.4340198Z Entering 'third_party/onnx-tensorrt' 2022-12-01T10:47:43.4374625Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-12-01T10:47:43.4412212Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-12-01T10:47:43.4445194Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-12-01T10:47:43.4477476Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-12-01T10:47:43.4516199Z Entering 'third_party/pocketfft' 2022-12-01T10:47:43.4550231Z Entering 'third_party/protobuf' 2022-12-01T10:47:43.4587316Z Entering 'third_party/protobuf/third_party/benchmark' 2022-12-01T10:47:43.4620376Z Entering 'third_party/protobuf/third_party/googletest' 2022-12-01T10:47:43.4655404Z Entering 'third_party/psimd' 2022-12-01T10:47:43.4689448Z Entering 'third_party/pthreadpool' 2022-12-01T10:47:43.4723737Z Entering 'third_party/pybind11' 2022-12-01T10:47:43.4757828Z Entering 'third_party/python-enum' 2022-12-01T10:47:43.4794233Z Entering 'third_party/python-peachpy' 2022-12-01T10:47:43.4828050Z Entering 'third_party/python-six' 2022-12-01T10:47:43.4861998Z Entering 'third_party/sleef' 2022-12-01T10:47:43.4896523Z Entering 'third_party/tbb' 2022-12-01T10:47:43.4932574Z Entering 'third_party/tensorpipe' 2022-12-01T10:47:43.4968486Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-12-01T10:47:43.5001695Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-12-01T10:47:43.5034420Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-12-01T10:47:43.5067910Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-12-01T10:47:43.5101027Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-12-01T10:47:43.5137134Z Entering 'third_party/zstd' 2022-12-01T10:47:43.5184364Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-12-01T10:47:43.5210752Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5217978Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-12-01T10:47:43.5249422Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-12-01T10:47:43.5501944Z Entering 'android/libs/fbjni' 2022-12-01T10:47:43.5521137Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5547729Z Entering 'third_party/FP16' 2022-12-01T10:47:43.5567726Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5593996Z Entering 'third_party/FXdiv' 2022-12-01T10:47:43.5614365Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5640366Z Entering 'third_party/NNPACK' 2022-12-01T10:47:43.5660420Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5688599Z Entering 'third_party/QNNPACK' 2022-12-01T10:47:43.5708360Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5734841Z Entering 'third_party/VulkanMemoryAllocator' 2022-12-01T10:47:43.5753816Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5781095Z Entering 'third_party/XNNPACK' 2022-12-01T10:47:43.5800867Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5840023Z Entering 'third_party/benchmark' 2022-12-01T10:47:43.5859950Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5887250Z Entering 'third_party/cpuinfo' 2022-12-01T10:47:43.5907131Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5933601Z Entering 'third_party/cub' 2022-12-01T10:47:43.5953904Z http.https://github.com/.extraheader 2022-12-01T10:47:43.5982189Z Entering 'third_party/cudnn_frontend' 2022-12-01T10:47:43.6002267Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6034145Z Entering 'third_party/cutlass' 2022-12-01T10:47:43.6054906Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6088017Z Entering 'third_party/eigen' 2022-12-01T10:47:43.6108114Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6136453Z Entering 'third_party/fbgemm' 2022-12-01T10:47:43.6156976Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6184296Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-12-01T10:47:43.6203608Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6229456Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-12-01T10:47:43.6248958Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6275487Z Entering 'third_party/fbgemm/third_party/googletest' 2022-12-01T10:47:43.6296060Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6322686Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-12-01T10:47:43.6341988Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6369697Z Entering 'third_party/flatbuffers' 2022-12-01T10:47:43.6389805Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6418154Z Entering 'third_party/fmt' 2022-12-01T10:47:43.6438166Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6464920Z Entering 'third_party/foxi' 2022-12-01T10:47:43.6485272Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6511277Z Entering 'third_party/gemmlowp/gemmlowp' 2022-12-01T10:47:43.6530979Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6557233Z Entering 'third_party/gloo' 2022-12-01T10:47:43.6577116Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6602862Z Entering 'third_party/googletest' 2022-12-01T10:47:43.6622585Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6649816Z Entering 'third_party/ideep' 2022-12-01T10:47:43.6669029Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6695073Z Entering 'third_party/ideep/mkl-dnn' 2022-12-01T10:47:43.6714994Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6742769Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-12-01T10:47:43.6762041Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6795011Z Entering 'third_party/ios-cmake' 2022-12-01T10:47:43.6814706Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6842000Z Entering 'third_party/ittapi' 2022-12-01T10:47:43.6861733Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6888211Z Entering 'third_party/kineto' 2022-12-01T10:47:43.6907344Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6933347Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-12-01T10:47:43.6952808Z http.https://github.com/.extraheader 2022-12-01T10:47:43.6979967Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-12-01T10:47:43.6999216Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7027235Z Entering 'third_party/nccl/nccl' 2022-12-01T10:47:43.7047216Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7073844Z Entering 'third_party/neon2sse' 2022-12-01T10:47:43.7094035Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7120505Z Entering 'third_party/nlohmann' 2022-12-01T10:47:43.7139981Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7168280Z Entering 'third_party/onnx' 2022-12-01T10:47:43.7187757Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7225932Z Entering 'third_party/onnx/third_party/benchmark' 2022-12-01T10:47:43.7246632Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7273834Z Entering 'third_party/onnx/third_party/pybind11' 2022-12-01T10:47:43.7293803Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7321992Z Entering 'third_party/onnx-tensorrt' 2022-12-01T10:47:43.7341579Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7368135Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-12-01T10:47:43.7387016Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7436027Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-12-01T10:47:43.7439175Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7466934Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-12-01T10:47:43.7486907Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7513333Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-12-01T10:47:43.7533766Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7564518Z Entering 'third_party/pocketfft' 2022-12-01T10:47:43.7585070Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7611135Z Entering 'third_party/protobuf' 2022-12-01T10:47:43.7630144Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7660839Z Entering 'third_party/protobuf/third_party/benchmark' 2022-12-01T10:47:43.7680458Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7707089Z Entering 'third_party/protobuf/third_party/googletest' 2022-12-01T10:47:43.7726835Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7754139Z Entering 'third_party/psimd' 2022-12-01T10:47:43.7774746Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7800540Z Entering 'third_party/pthreadpool' 2022-12-01T10:47:43.7820451Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7847816Z Entering 'third_party/pybind11' 2022-12-01T10:47:43.7867038Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7893340Z Entering 'third_party/python-enum' 2022-12-01T10:47:43.7912887Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7938532Z Entering 'third_party/python-peachpy' 2022-12-01T10:47:43.7959242Z http.https://github.com/.extraheader 2022-12-01T10:47:43.7985484Z Entering 'third_party/python-six' 2022-12-01T10:47:43.8004836Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8031047Z Entering 'third_party/sleef' 2022-12-01T10:47:43.8051168Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8077299Z Entering 'third_party/tbb' 2022-12-01T10:47:43.8097628Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8126466Z Entering 'third_party/tensorpipe' 2022-12-01T10:47:43.8145334Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8171719Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-12-01T10:47:43.8190884Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8217316Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-12-01T10:47:43.8236969Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8262394Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-12-01T10:47:43.8281424Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8307926Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-12-01T10:47:43.8328362Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8354108Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-12-01T10:47:43.8372907Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8401807Z Entering 'third_party/zstd' 2022-12-01T10:47:43.8421467Z http.https://github.com/.extraheader 2022-12-01T10:47:43.8654172Z Cleaning up orphan processes